What happens when you predict the winner of all 50 states in the presidential election? If you happen to write for the New York Times, you become famous. Indeed, you become a celebrity statistician. (Yes, it’s a real thing.) Nate Silver did just that in 2012. He correctly predicted the winner of each state and became famous. As Rachel Maddow of MSNBC put it, “You know who won the election tonight? Nate Silver.”
Nobody thinks this is more ridiculous than Nate Silver himself. In interviews after the election, he was quick to point out that an eighth grader could have predicted about 40 of the 50 states. The other swing states became increasingly clear as Election Day drew near, despite pundits on both sides predicting a nail-biter. Silver held his prediction in perspective and with humility.
Humility in prediction is the theme of Silver’s book. He describes a host of examples where predictions were made with inappropriately high levels of confidence. At times, as in the financial crisis, there were disastrous results. Silver sees this as a growing problem caused by the emergence of “big data.” Given an exponential increase in the availability of data, the ability to extract the signal from the noise becomes increasingly important, and failure to do so efficiently can have serious consequences. Complex systems, like the climate or the economy, are incredibly difficult to predict either because we lack the necessary data or because we lack a theory to make sense of the data. At no point does Silver suggest we should stop making predictions, he simply (and wisely) advises that we make and consume predictions with greater humility and understanding.
Continue reading “The Signal and the Noise by Nate Silver [Book Review]”