Well, it is the time of year when America’s greatest sporting event takes place. I speak of course about the race to determine which Super Bowl ad is the best. Over the years there have been many ways to accomplish this, but like so often happens in research today, the methods are flawed.
First there is the “party consensus method”. Here people gathered to watch the big game call out their approval or disapproval of various ads. Beyond the fact that the “sample” is clearly not representative, this method has other flaws. At the party I was at we had a Nationwide agent, so criticism of the “dead kid” ad was muted. This is just one example of how people in the group can influence each other (anyone who has watched a focus group has seen this in action). The most popular ad was the Fiat ad with the Viagra pill…not because it was perhaps the favorite, but because parties are noisy and this ad was largely a silent picture.
Second, there is the “opinion leaders” method. The folks who have a platform to spout their opinion (be it TV, YouTube, Twitter or Facebook) tell us what to think. While certainly this will influence opinions, I don’t think tallying up their opinions really gets at the truth. They might be right some of the time, but listening to them is like going with your gut…likely you are missing something.
Third, there is the “focus group” approach. In this method a group of typical people is shuffled off to a room to watch the game and turn dials to rate the commercials they see. So, like any focus group, these “typical” people are of course atypical. In exchange for some money they were willing to spend four hours watching the game with perfect strangers. Further, are focus groups really the way to measure something like which is best? Focus groups can be outstanding at drawing out ideas, providing rich understandings of products and so on, but they are not (nor are they intended to be) quantitative measures.
The use of imperfect means to measure quantitative problems is not unique to Super Bowl ads. I’ve been told by many clients that budget and timing concerns require that they answer some quantitative questions with the opinions of their internal team, or their own gut or qualitative research. That is why we developed our agile and rigorous tools, including Message Test Express™ (MTE™).
MTE™ uses our proprietary Bracket™ method and a standardized approach to deliver quantitative results in a week's time and starting from $9,900. So, I decided to see if it would work for Super Bowl ads. Monday morning we launched a survey which measured the relative merit of 36 Super Bowl ads. Respondents were asked to choose their most and least entertaining from groups of 3 ads. The “winners” then played off against each other until the respondent had picked their favorite. Through the use of Hierarchical Bayesian Estimation (the same math that makes Max-Diff or Discrete Choice conjoint work) we were able to produce utilities (which divide 100 points among all the ads tested) and with that quantitatively determine which ad was the most entertaining and how much better it was than the next ad.
The winner was the Budweiser Puppy Ad, which is no great surprise (if you’ve watched the coverage). The margin of victory, however, might be. As you can see in the chart above, Budweiser’s ad got over four times as high a utility score as second place Fiat/Viagra did. In fact, Budweiser’s ad was the favorite of roughly one in three respondents and ranked high for the vast majority. The ad did much better among women than it did among men (though it won with both groups) and scored well with both beer drinkers and non beer drinkers.
The Toyota “Being a Dad is more than being a father” ad came in fourth, but importantly it scored far better among those looking to buy a car in the coming months than among those who do not. Easy enough to forget that the point of these ads is to sell product!
I found it interesting that the Snickers Brady Bunch ad scored well with all age groups under 65. I had theorized that it would score better with those over 50 (who saw it as kids), but the show somehow still resonates 40 years after it went out of production. Only those too old to have “appreciated” the show when it first aired ranked the ad low.
One of my favorites, which hasn’t shown up on many lists, was the Doritos Airplane ad, which our study ranked 5th. Perhaps I’ve just been traveling a lot lately but it spoke to me.
The Victoria's Secret ad ranked 6th and suprisingly it scored equally among both men and women.
Finally, an ad that I’ve heard a lot of commentators and Facebook posts talk about is the Squarespace “Dreaming with Jeff Bridges Ad” With apologies to the “Dude”, our representative sample ranked it as 35th (out of 36 tested).
We’ll be digging further into the results of this research over the coming days so stay tuned for more and in the meantime if you would like to know more about the methodology or the results, feel free to reach out.
|2015 Super Bowl Ads Ranked from the Most to Least Entertaining|
|#1 Budweiser - Lost Dog|
|#2 - Fiat 500X Crossover - Viagra in the gas tank|
|#3 - Snickers - Brady Bunch - You're not you when you're hungry|
|#4 - Toyota - Being a dad is more than being a father|
|#5 - Doritos - Airplane seat saver with mom and baby|
|#6 - Victoria's Secret - I'm in the mood for love, models posing|
|#7 - esurance - 2 ads: Lindsay Lohan and Walter White|
|#8 - Bud Light - Life-size Pac-Man game|
|#9 - BMW i3 - Katie Couric and Bryant Gumbel|
|#10 - Always - What it means to be a girl|
|#11 - Clash of Clans - Liam Neeson doesn't like to lose|
|#12 - Coca-Cola - Spilled Coke changes the internet to make the world happy|
|#13 - Doritos - When pigs fly|
|#14 - Dove Men+Care - What it means to be a dad today|
|#15 - Mercedes - Tortoise and the hare fable|
|#16 - Nationwide - Mindy Kaling is invisible, with Matt Damon|
|#17 - Avocado from Mexico - First draft ever|
|#18 - McDonald's - Pay by calling your loved ones|
|#19 - Microsoft - 2 ads: Braylon's prosthetic legs and Estellas' brilliant bus|
|#20 - Skechers - Pete Rose isn't supposed to be in the hall|
|#21 - Skittles - Arm wrestlers battle over the last Skittle|
|#22 - Nissan - Race car driver and son - Cat's in the cradle|
|#23 - Dodge - Centenarians give advice|
|#24 - Loctite Glue - Everyday people dancing|
|#25 - Turbo Tax - Boston Tea Party|
|#26 - T-Mobile - Chelsea Handler and Sara Silveramn one-up each other|
|#27 - Kia Sorento - Perfect getaway vehicle with Pierce Brosnan|
|#28 - Wix.com - F. Harris, L. Allen, E. Smith, T. Owen, B. Favre build websites|
|#29 - Mophie - When your phone dies, God knows what can happen|
|#30 - T-Mobile - Kim Kardashian West introducing data stash|
|#31 - GoDaddy - Guy working hard misses the big game|
|#32 - Chevy Colorado - What if your TV went out...use wifi in your vehicle|
|#33 - Lexus NX Turbo Hybrid - Make some noise in a glamorous parking lot|
|#34 - Carnival Corporation - Come back to the sea|
|#35 - Squarespace - Dreaming with Jeff Bridges|
|#36 - Cure Auto Insurance - Deflated balls|
Rich brings a passion for quantitative data and the use of choice to understand consumer behavior to his blog entries. His unique perspective has allowed him to muse on subjects as far afield as Dinosaurs and advanced technology with insight into what each can teach us about doing better research.