An issue that comes up quite a bit when doing research is the proper way to frame questions. In my last blog I reported on our Super Bowl ad test in which we surveyed viewers to rank 36 ads based on their “entertainment value”. We did a second survey that framed the question differently to see if we could determine which ads were most effective at driving consideration of the product…in other words, the ads that did what ads are supposed to do!
As with life, framing, or context, is critical in research. First off, the nature of questions is important. Where possible the use of choice questions will work better than say rating scales. The reason is that consumers are used to making choices...ratings are more abstract. Techniques like Max-Diff, Conjoint (typically Discrete Choice these days) or our own proprietary new product research technique Bracket™ get at what is important in a way that ratings can’t.
Second, the environment you create when asking the question must seek to put consumers in the same mindset they would be in when they make decisions in real life. For example, if you are testing slogans for the outside of a direct mail envelope, you should show the slogans on an envelope rather than just in text form.
Finally, you need to frame the question in a way that matches the real world result you want. In the case of a direct mail piece, you should frame the question along the lines of “which of these would you most likely open?” rather than “which of these slogans is most important?”. In the case of a Super Bowl ad (or any ad for that matter), asking about entertainment value is less important than asking about things like “consideration” or even “likelihood to tell others about it”.
So, we polled a second group of people and asked them “which one made you most interested in considering the product as advertised?” The results were quite interesting.
In both cases, the Budweiser Puppy ad won by a landslide. In fact, it did even better for consideration AND it did well among both current beer drinkers and non-drinkers. In other words, the ad had broad reaching appeal AND drove consideration even more broadly. Congratulations to Budweiser.
The Fiat/Viagra ad came in second for “entertainment value” but a distant 5th for “consideration”. Easy enough to argue that this could have been driven by the fact that the market for cars is far more limited than the market for beer drinkers.
But what if we pick a category that most people use such as “junk food”? I picked out six ads for McDonalds, Snickers, Coca Cola, Skittles and Doritos (2 different ads) to see how they ranked for both “Entertainment” and “Consideration”. The ads and their “entertainment value” ranks were:
As this list shows, the Snickers ad was far and away the most entertaining, while the McDonalds ad was only ranked fifth…barely ahead of sixth place Skittles.
But, if we change the framing to “consideration” the McDonalds ad leaps to number one (see chart below). Snickers by contrast dropped quite a bit and though it was in second place, it was a distant second, barely outpacing the other ads. None of the others changed much, but the Doritos “Pigs Fly” ad improved while the “Airplane Seat” ad did worse such that their positions switched. Skittles did poorly under both framing statements.
As a reminder, both tests were done using our Message Test Express™ product. It allows us to take long and complex list of 6 to 36 items and determine which ones will work best and by how much they are better than the next item on the list. It uses our proprietary tool Bracket™ which breaks the list into random groups of three items (different groupings for each respondent) and to pick their most and least favorite from each group and then pick among the winners in a similar fashion. It is very engaging and because it uses a standard approach it allows things to be turned around quite quickly (the Super Bowl ad survey was done in 48 hours).
So, in my view, Budweiser deserves great credit for producing an ad that captivated, was talked about AND helped to sell the brand. McDonalds, also deserves credit. While their ad came in 18th place out of 36 overall for “entertainment value” it came in second for “consideration” and it did so without the use of a puppy!
PS.Two other ads worthy of mention for doing better with regards to consideration were the Dove Men+Care (what it takes to be a Dad) and the Always (What it means to run like a girl). They ranked only 14th and 10th for “entertainment value” (not surprising given the deep message they were trying to convey), but they were 3rd and 4th when it came to consideration. Again…without a puppy.
Rich brings a passion for quantitative data and the use of choice to understand consumer behavior to his blog entries. His unique perspective has allowed him to muse on subjects as far afield as Dinosaurs and advanced technology with insight into what each can teach us about doing better research.