There is nothing like being right to make an impression. After calling the majority of congressional districts in the 2012 US election Nate Silver enjoyed his 15 minutes of fame. Before his election prediction I was only dimly aware of Nate Silver. I knew he worked for the New York Times, but that’s no longer an indicator of excellence or even sanity. Heck, even Nobel Prizes no longer guarantee excellence or sanity. Obama, vain narcissist that he is, was embarrassed by the dolts on the Nobel Peace Prize committee that confused existence for accomplishment.
It wasn’t Silver’s employer that led me to his book; it was his stint as a serious poker player that told me he wasn’t a standard NYT brain-dead progressive. Progressives do not bet with their own money! They bet with other people’s money. Anyone that puts their money where their mouth is is worth a hearing and Silver is worth a hearing
The Signal and the Noise is a series of essays about making predictions. It won’t surprise anyone to learn that some fields suffer poor, dare I say idiotic, predictors. Economists and partisan policy wonks are among the worst. Silver’s statistics show many of these people are clueless ideologues or cynical liars. They’re not even reliable contra-indicators. If only Nancy Pelosi, that Botox saturated crone, was consistently wrong, rather than randomly moronic, we might profit from her emissions.
As bad as some predictors are it’s not all bleak. Meteorologists have dramatically improved their forecasts. A few decades ago it was anyone’s guess where hurricanes would hit land. Now it’s possible to forecast landfalls within a hundred miles two days in advance. The weather service called Katrina before it hit New Orleans. It’s too bad so many ignored the warning.
One of the best sections in The Signal and the Noise deals with the dangers of “over-fitting”, over-fitting occurs when a model ends up modeling noise instead of signal. Over-fitting is an egregious statistical error but human beings are evolved over-fitters. If you “predict” the wind is shaking a bush and it’s a tiger you’re cat food. If you predict a tiger is shaking a bush and it’s the wind you have a bad hair day. Evolution favors the latter. If life is short, nasty and brutish, it pays to over-fit immediate dangers. This is not the case when over-fitting tells you something is highly unlikely when it isn’t.
Silver makes a good case for the Fukushima nuclear disaster going down in history as a classic case of the dangers of over-fitting. Earthquakes are currently unpredictable. Silver goes over the history of earthquake prediction and it’s sobering. Forecasts made by 21st century geophysicists, armed with petaflop supercomputers, are only marginally better than simple historical means. This is a tough scientific problem made orders of magnitude harder by the difficulty of collecting data. We cannot directly measure stresses twenty kilometers underground. Hence the data feeding earthquake models are at best approximate and incomplete. This is unfortunate because models based on sketchy data are essentially conspiracy theories without black helicopters. You won’t find many geophysicists making short-term Vegas bets on the output of their earthquake models.
This doesn’t mean that earthquakes are random or lack order. Earthquakes are remarkably orderly over geological timescales. They eerily fit a power-law distribution. This excellent fit makes it possible to pick any point on the Earth and compute an earthquake probability. Such probabilities were computed for the seas near Fukushima but the earthquake model used was over-fitted and it dramatically underestimated the likelihood of magnitude 9 earthquakes. The Fukushima model had been “tweaked” to echo the fact that rare magnitude 9 earthquakes had not been observed near Japan in centuries. Instead of following a nice linear log-log plot the Fukushima plot was “bent” and the bend led to the assumption that it wasn’t necessary to plan for magnitude 9 earthquakes and potentially huge tsunamis. Here model over-fitting led to seawall over-topping. This is not your average stats 101 screw-up.
Looking back it’s easy to see where people went wrong. Maybe evil crony capitalists, bent on saving a few yen on concrete, conspired to sabotage earthquake models before submitting low ball Fukushima seawall bids. Doesn’t Lex Luthor do this every other day? Here it’s not necessary to invoke super-villains, good old fashioned short-term thinking, fortified with professional hubris, is all that’s required. Silver makes it abundantly clear that prediction, “especially about the future”, is hard but not necessarily hopeless. This is an excellent book for both lovers and haters of statistics.
Analyze the Data not the Drivel 