Business analysts would do well to pay attention to the scrutiny that election prediction models engender. While the readouts from a customer analytics project won’t rival the passions generated by a close contest for the White House, there are two important takeaways from the political fray. First, the audience for your predictions needs to understand the terms you use to communicate your findings. Second, you need to be ready to defend your model against loud criticism.
Two high-profile models provide a view of how very different approaches to the same predictive problem play out in public. Statistician and writer Nate Silver, of The New York Times’ FiveThirtyEight blog, conducts a meta-analysis of many polls and economic data to derive forecasts of likely outcomes. Allan Lichtman, an American University historian and author of the book Keys to the White House, has developed a model that projects the popular vote winner by examining the incumbent party’s performance in the White House based on 13 conditions, including the U.S. economy, foreign affairs, social unrest, political scandals and the makeup of the challenger.
While both analysts have reason to brag—Silver correctly called voting preferences in 49 of 50 states in 2008, and Lichtman’s model, developed in 1981, has predicted the popular vote winner in every election since 1984—the pair have different communication styles.
Lichtman issued his prediction that President Barack Obama would win the popular vote in his reelection bid more than two years before a vote was cast—in July 2010. That was when a broken BP oil well was gushing crude into the Gulf of Mexico, American forces were still fighting in Iraq and the jobless rate was 9.5 percent. In this model, the incumbent party wins when eight or more of the 13 keys fall in their favor. Obama had nine in his favor then, and Lichtman said in a newspaper column on Oct. 5, that his assessment gave the president 10 in his favor.
Silver publishes the daily results of simulations using variations of the data to account for discrepancies between national and state polls, and the effects of economic indicators on the presidential race. As of Nov. 5, Silver’s model gave Obama an 86.3 percent chance of winning re-election, with a squeaker popular vote victory of 50.6 percent over former Massachusetts Gov. Mitt Romney’s 48.5 percent. (Note that Silver frames his prediction in terms of probability, not certainty.) Silver noted that Obama prospects rose after Hurricane Sandy hit the East Coast, but that the president had been gaining in the polls even before the storm.
The Noise Surrounding the Signals
That an analyst would face scrutiny for the results he presents is a natural condition in the business world; executives should kick the tires on reports. But the controversy around Silver’s projections in the weeks leading up to the election is instructive, and shows why it pays to lay the groundwork for an analysis based on probability and statistics.
For example, Joe Scarborough, the host of “Morning Joe” on MSNBC, took direct aim at Silver’s projections that Obama was favored to win. “Anybody that thinks that this race is anything but a tossup right now is such an ideologue, they should be kept away from typewriters, computers, laptops and microphones for the next 10 days, because they’re jokes,” Scarborough said. (The TV host is not the only one criticizing Silver’s analysis, as U.S. News points out.)
With his blog receiving widespread media coverage and a new bestselling book The Signal and the Noise to discuss, Silver has had opportunities to defend himself. And he has used these opportunities to suggest that Scarborough, like many people, doesn’t understand the math terms he uses. It helps to know that probability is different than certainty, for one thing. Silver said this on “The Charlie Rose Show” when asked to explain what “separating signal from noise” means:
“It means learning how to work with information to perceive where the real value is there. I think sometimes people tend to overrate the value at the margin of new data that they see. Where they see so many economic statistics, so many polls, people tend to go a little bit haywire and change their views about the economy or the election every few minutes. In fact it’s usually better to realize that any one poll is not all that good an indicator, just like any one economic statistic. But if you take them all together and they all show the same story, that 9 out of 10 of the polls in Ohio, not every one, but 9 out of the last 10, show Obama ahead—that becomes more meaningful.”
One Model Against Another
As the controversy over Silver’s projections shows, analysts have to be ready to defend their methods. It turns out that Lichtman and Silver have gone at it with each other. Last summer, Silver published a critique of Lichtman’s 13 keys model on his blog, calling some of the criteria subjective and noting that it fails to account for margins of victory.
Lichtman subsequently published a defense on the FiveThirtyEight blog, writing that his model is not designed to predict the margins of the results, and adding: “The amount of subjectivity is far less than meets the eye, given the careful definition of each key in the published material, the record of how each key was turned in the 38 elections in 1860 to 2008, and the successful predictions from 1984 to 2008.”
His bigger point, Lichtman writes, is that a president’s governing matters more than campaigns: “It suggests that the American electorate makes reasoned, pragmatic decisions in presidential elections and is not manipulated by the pollsters, the admen and the consultants,” he writes.