Superforecasting: How Ordinary People Outsmarted Intelligence Agencies

Superforecasting: The Art and Science of Prediction is a book by Philip E. Tetlock Dan Gardner that explains how one can become an expert at forecasting and apply this skill to any situation in life. In the chapter titled “Superforcasters” the author starts with an example of a judgement of all the intelligence agencies that Iraq had weapons of mass destruction and how it turned out to be totally wrong. Even the agencies spending billions of dollars and having a lot of data at their disposal could make that mistake. Then the author talks about the study, Good Judgment Project, which was basically a tournament organized by IARPA where they asked people to give their forecasts on real-life scenarios from the world, for example “Will an outbreak of H5N1 in China kill more than ten in the next six months?”. The result of this study they did find some superforecasters who had the most accurate results among other forecasters in the study.

Interestingly, some sources reported that their forecasts were even more accurate than the analysts at the agencies. But the author also argues whether it was only luck that they got it right or it was their skills. He explains it by the regression to mean concepts that conclude that the performance of the people who were superforecasters should get close to the average in the second round. But that did not happen, and they got even better. Then in the next chapter, it is explained that the people in the superforecasters had better intelligence levels than the general population in the United States, so it is possible they were super smart. In my opinion, the author’s use of real-life examples to articulate his concepts was really good. Some statistical knowledge was shared, and it was a bit complicated, but diagrams and examples were used to explain these too. Also, I think Fermi’s question about the piano players in Chicago was a great example. It explained clearly how you can break down a question into multiple other questions if you don’t have the answer. And then, by answering the sub-questions, you can make an educated guess. He also described some of the methods that superforecasters use in making their judgements which were well explained. John Horgan whoDirects the Center for Science Writings said that if we can learn the critical thinking of the superforecasters mentioned in the book we can learn to make better predictions and forecasts. And I feel the same way because the analysis of the author shows that it was not mere luck that these superforecasters emerged from the study.

References:

https://blogs.scientificamerican.com/cross-check/can-we-improve-predictions-q-a-with-philip-superforecasting-tetlock/