Was the election outcome a surprise for you? It wasn’t for me.
In some ways election night was quite boring. And I blame Nate Silver, Sam Wang and others who predicted the outcome with such stunning accuracy that (at least for me) the drama was completely missing. While conventional pundits and partisans were making all kinds of predictions ranging from “Toss-up” to “Romney landslide”, a group of analysts (nerds, if you choose) were quietly predicting that Obama had a small but consistent and predictable lead. Turns out they were spot-on in their predictions (and were predictably smeared by vested interests).
In my last post I talked about Nate Silver and the approach he uses. This time I want to draw your attention to another analyst, Sam Wang of the Princeton Election Consortium. He is a neuroscientist who has been forecasting for the last three presidential election cycles and has been doing a remarkably good job of it. He nailed the Electoral College vote in 2004 and missed by just one in 2008. How did he do this time? Well, he had two predictions. One of them (based on his median estimator) was 303 for Obama, which is where the tally currently stands, subject to Florida being officially called. The second one (based on his modal estimator) was 332 for Obama which is where the tally is likely to end up if/when Obama wins Florida. Excellent calls whichever way you look at it, given the extremely close race in Florida.
What makes Wang an even more interesting case for market researchers is that, he uses ONLY (state-level) survey data. Unlike Nate Silver he does not mix in economic data for his prediction models, nor makes any adjustments (such as for “house-effects”). He asserts that the information from economic data is already “priced” into survey data and is thus not needed. So he feels that in that respect Nate Silver’s model is unnecessarily complicated, although it can be useful in information-poor situations like unheralded House races or early stage Presidential predictions. Wang takes every survey that is available, unlike Silver who excludes some. His main trick is to use medians rather than mean to average the surveys, thus eliminating the impact of outliers – a trick that we researchers know well. But this simple process yields remarkably good results. And as Silver does, he also runs simulations to calculate probabilities of victory in each state (explained here).
So what we have here is quite interesting. An election that was very close and that most neutral mainstream pundits (especially those who insisted on looking at only national polls) were saying was too close to call. But primarily (and even exclusively) using survey data, intelligent analysts were able to predict almost exactly what was going to happen. Nate Silver called every single one of the 50 states correctly. And interestingly enough there was even a way to validate his probabilities. All you had to do was look at the clock. The states that he had high probabilities for were called early and the ones he had as a lean were called later. Florida was the closest one (with a 50%-55%) probability for Obama and is the last one to be called. People often say prediction probabilities cannot be validated because elections are run only once (unlike simulations), but surely this provides a nice way to validate it?
Given all this, the inevitable question is what can we market researchers learn?
Surveys are very useful…
Given the explosion of new methodologies in research there has been a tendency to put down surveys. But as the election results show, they contain very useful and rich information and are a wonderful way to make accurate predictions. Most of the surveys used for election forecasting are in fact done the old-fashioned way using telephones.
…when done right
While surveys may be old-fashioned they face new problems every day. Problems like decreasing response rates, increasing number of cellphone-only (and other unreachable) households present great challenges for survey researchers. But rather than throwing up their hands, researchers have taken more and more steps to compensate for these problems and have found success.
Strength in numbers also implies the opposite
If these last two presidential election cycles taught us anything, it should be that aggregating surveys is very valuable. Any one survey can and will be misleading at least some o f the time, while the aggregate can be a steady and reliable predictor. But aggregation is much easier to do in presidential elections where dozens of surveys are released. What about market research client studies where a single survey is done and that too with a small sample size (often less than 500)? For me, the biggest lesson is to think of single surveys primarily as directional indicators. Too often we tend to split hairs, run endless significance tests and try to read signals where there is too much noise. In single surveys (whether using crosstabs or conjoint) it is best not to try to interpret the noise.
Surveys in election forecasting worked so well because they were used for an appropriate purpose – to understand what a large group of people intended to do about a specific issue. They can be as valuable when used in a similar manner in market research (i.e.) for quantitative generalization. There are a variety of other methods (especially qualitative) that have great strengths in explorative and deep dive exercises. When the methods are used appropriately and as complements we are likely to get excellent results.
So take heart survey researchers, we do have a place in this rapidly changing market research world!