I read an interesting story about a survey done to determine if people are honest with pollsters. Of course such a study is flawed by definition (how can we be sure those who say they always tell the truth, are not lying?). Still, the results do back up what I’ve long suspected…getting at the truth in a survey is hard.
The study indicates that most people claim to be honest, even about very personal things (like financing). Younger people, however, are less likely to be honest with survey takers than others. As noted above, I suspect that if anything, the results understate the potential problem.
To be clear, I don’t think that people are just being dishonest for the sake of being dishonest….I think it flows from a few factors.
First, some questions are too personal to answer, even on a web survey. With all the stories of personal financial data being stolen or compromising pictures being hacked, it shouldn’t surprise us that some people might not want to answer some kinds of questions. We should really think about that as we design questions. For example, while it might be easy to ask for a lot of detail, we might not always need it (income ranges for example). To the extent we do need it, finding ways to build credibility with the respondent are critical.
Second, some questions might create a conflict between what people want to believe about themselves and the truth. People might want to think of themselves as being “outgoing” and so if you ask them they might say they are. But their behavior might not line up with reality. The simple solution is to ask questions related to behavior without ascribing a term like “outgoing”. Of course, it is always worth asking it directly as well (knowing the self image AND behavior could make for interesting segmentations variables for example)....
Rita’s Italian Ice is a Pennsylvania-based company that sells its icy treats through franchise locations on the East Coast and several states in the Midwest and West.
Every year on the first day of spring, Rita’s gives away full-size Italian ices to its customers. For free. No coupon or other purchase required. It’s their way of thanking their customers and launching the season (most Rita’s are only open during the spring and summer months).
Wawa, another Pennsylvania company, celebrated 50 years in business with a free coffee day in April.
Companies are giving their products away for free! What a fantastic development for consumers! I patronize both of these businesses, and yet, on their respective free give-away days, I didn’t participate. I like water ice (Philadelphia’s term for Italian ice) and I really like coffee. So what’s the problem?
In the case of Rita’s, the franchise location near me has about 5 parking spots, which on a normal day is too few. I was concerned about the crowds. On the Wawa give-away day, I forgot about it as the day wore on. That made me wonder what other people do when they learn that retailers are giving away their products. So, having access to a web-based research panel (a huge perk of my job), I asked 485 people about it. And here are the 4 things I learned:...
In my previous post I applauded Matthew Futterman’s suggestion that two key changes to baseball’s rules will produce a shorter, faster-paced game, one that will attract younger viewers. While I may not be that young, I’m certainly on-board with speeding up the game. I believe that faster-paced play will lead to greater engagement, and greater engagement will lead to greater enjoyment.
In some sense this is similar to our position on marketing research methods. We want to engage our respondents because the more focused on the task they become, the more considered their responses will be. One of our newer tools, Bracket,TM allows respondents to prioritize a long list of items in a tournament-style approach. Bracket™has respondents make choices among items, and as the tournament progresses the choices become more relevant (and hopefully more enjoyable).
Meanwhile, back to baseball. The rule changes Futterman suggests are very simple ones:
Once batters step into the box, they shouldn't be allowed to step out. Otherwise it's a strike.
If no one is on the base, pitchers get seven seconds to throw the next pitch. Otherwise it's a ball....
As most anyone living on the East Coast can attest, the winter of 2013-2014 was, to put it nicely, crappy. Storms, outages, freezing temperatures…. We had a winter the likes of which we haven’t experienced in a while. And it wasn’t limited to the East Coast – much of the US had harsher conditions than normal.
Here in the office we did a lot of complaining. I mean a lot. Every day somebody would remark about how cold it was, how their kids were missing too much school, how potholes were killing their car’s suspension… if there was a problem we could whine about, we did.
Now that it’s spring and we’re celebrating the return of normalcy to our lives, we wonder… just what was it about this past winter that was the absolute worst part of it? Sure, taken as a whole it was pretty awful, but what was the one thing that was the most heinous?
Fortunately for us, we have a cool tool that we could use to answer this question. We enlisted the aid of our consumer panel and our agile and rigorous product Message Test Express™ to find the answer. MTE™ uses our proprietary Bracket™ tool which takes a tournament approach to prioritizing lists. Our goal; find out which item associated with winter was the most egregious.
Our 200 participants had to live in an area that experiences winter weather conditions, believe that this winter was worse or the same as previous winters, and have hated, disliked or tolerated it (no ski bums allowed)....
Market Researchers are constantly being asked to do “more with less”. Doing so is both practical (budgets and timelines are tight) and smart (the more we ask respondents to do, the less engaged they will be). At TRC we use a variety of ways to accomplish this from basic (eliminate redundancies, limit grids and the use of scales) to advanced (use techniques like Conjoint, Max-Diff and our own Bracket™ to unlock how people make decisions). We are also big believers in using incentives to drive engagement and with more reliable results. That is why a recent article in the Journal of Market Research caught my eye.
The article was about promotional lotteries. The rules tend to be simple, “send in the proof of purchase and we’ll put your name in to a drawing for a brand new car!." The odds of winning are also often very remote which might make some not bother. In theory, you could increase the chances of participation by offering a bunch of consolation prizes (free or discounted product for example). In reality, the opposite is true.
One theory would be that the consolation prizes may not interest the person and thus they are less interested in the contest as a whole. While this might well be true, the authors (Dengfeng Yan and A.V.Muthukrishnan) found that there was more at work. Consolation prizes offer respondents a means to understand the odds of winning that doesn’t exist without them. Seeing, for example, that you have a one in ten million chance of winning may not really register because you are so focused on the car. But if you are told those odds and also the much better odds of winning the consolation prize you realize right away that at best chances are you will win the consolation prize. Since this prize isn’t likely to be as exciting (for example, an M&M contest might offer a free bag of candy for every 1000 participants), you have less interest in participating.
Since we rely so heavily on incentives to garner participation, it strikes me that these findings are worthy of consideration. A bigger “winner take all” prize drawing might draw in more respondents than paying each respondent a small amount. I can tell you from our own experimentation that this is the case. In some cases we employ a double lottery using our gaming technique Smart Incentives™ tool (including in our new ideation product Idea Mill™ ). In this case, the respondent can win one prize simply by participating and another based on the quality of their answer. Adding the second incentive brings in additional components of gaming (the first being “chance”) by adding a competitive element.
Regardless of this paper, we as an industry should be thinking through how we compensate respondents to maximize engagement....
I'm 5'5" tall, and my neck hurts. No, not all the time, just when I fly. And why does my neck hurt? Simple. Most economy-class seats have stationary bump-out head rests. Now, I’m not quite sure why these headrests bump out like this, since travelers of differing heights will surely experience them differently. My particular problem is that I’m just tall enough to get the back of my head to the bump-out...which means that when I place my head against it it pushes my entire head forward and down. So my neck hurts.
My best solution so far is to buy a neck pillow and wear it backwards. This helps to prop up my chin, but the bump-out and the neck pillow are in a constant battle for supremacy, and usually the bump-out wins.
So what does this have to do with research? Plenty. I wanted to know if other airline passengers would get excited by the prospect of an adjustable headrest to accommodate their needs. Of course, everyone would be happy with something designed just for them, so we tested it alongside other potential cabin improvements to see where it would land.
|CABIN IMPROVEMENTS TESTED:|
|Adjustable headrest to accommodate any size traveler|
|Denser seat cushions for added comfort|
|Folding foot rests to elevate your feet|
|Lumbar support built into the seat backs|
|More leg room than in a standard exit row seat|
|Roomier seats - 2 inches wider than most domestic airlines|
|Seats recline 5 degrees further than other airlines' seats|
|Tray tables with non-slip surface - better for gripping beverages|
We surveyed our intrepid online research panelists, limited the pool to those who fly, and applied our Message Test Express™technique, which is a tournament-style method of having respondents make choices from a list of items. MTE delivers rank ordering with a numeric value so you can see not just how they ranked, but how close they were in the order....
I read an astounding fact this week, “More Indians have used a mobile phone than a toilet”. It seemed absurd to me that a relatively new technology would outpace an old (and very useful) one. I came to realize that the absurdity was mainly due to the fact that I couldn’t imagine a world without either device and it struck me that this is an example of what ails the market research world.
The fact is that Indians have not chosen the cell phone over indoor plumbing. The former is widely available (because cell phone infrastructure is relatively easy to build) and the latter is not. So it wasn’t a choice of toilets over telcom, it was a choice of having a cell phone or not having one. Those who got the phones have begun to find uses for it that go far beyond the obvious. For example, fishermen call in while at sea to find out which port is offering the best price for their catch, thus maximizing their profits.
In Market Research we are often blinded by our experience. Instead of viewing new market research technology for its potential, we view it through the lens of what we know. When web data collection arrived, many didn’t see the opportunities it offered and instead defensively dismissed it as being inferior to existing methods and only offered the benefits of being “cheap and fast”. After more than a decade, it amazes me how many still hold this belief.
For years now, my colleague Jessica would solicit donations to the American Cancer Societythrough its annual Daffodil Days® campaign. Each year I'd give Jessica my donation and a few weeks later I'd receive 10 daffodil buds. I'd arrange them in a vase in my office and watch as they opened up into beautiful blooms over the course of a few days. And in doing so I'd be reminded that my donation is being used to find ways to eradicate cancer and help people in need.
It was announced that this year would be the final year for Daffodil Days®.
I have to admit, my first thought was not, "how will I donate to ACS now?" My first thought was that something was being taken away from me! Which, of course, irritated me. My second thought was that I'll have to look for another way to get daffodil buds next spring. And then it dawned on me that by cancelling the daffodils promotion, the ACS could be losing a long-time supporter.
Businesses are faced with product optimization decisions all the time – what will happen if I remove a product, service or distribution channel from the market? Will customers be lost? What will the short- and long-term effects be?
"Scientists always want to be wrong in their theories. They always want to be surprised."
He went on to explain that surprise is what leads to new discoveries whereas simply confirming a theory does not. I can certainly understand the sentiment, but it is not unusual for Market Research to confirm what a client already guessed at. Should the client be disappointed in such results?
I think not for several reasons.
First, certainty allows for bolder action. Sure there are examples of confident business people going all out with their gut and succeeding spectacularly, but I suspect there are far more examples of people failing to take bold action due to lingering uncertainty. I also suspect that far too often overconfident entrepreneurs make rash decisions that lead to failure.
Second, while we might confirm the big question (for example in product development pricing research we might confirm the price that will drive success) we always gather other data that help us understand the issue in a more nuanced way. For example, we might find that the expected price point is driven by a different feature than we thought (in research speak, that one feature in the discrete choice conjoint had a much higher utility score than the one we thought was most critical)....
Okay, so it wasn’t really just the two of us – there were a few hundred others involved. Still, it was a very memorable evening that I think is worth sharing.
The day started innocently enough. I was heading out to Yale for a guest lecture in the MBA Marketing Research class taught by Jiwoong Shin as I have done for several Spring semesters now. I like this trip a lot as it allows me to catch up with many of my friends in the Yale Marketing Department. One of those is Shane Frederick and I had emailed him to see if he was around. He replied asking if I was attending Kahneman’s lecture. I had no idea that Daniel Kahneman, Nobel Prize winner and godfather of behavioral economics was giving a lecture there. The day was already getting better! I quickly changed my Amtrak ticket to a later time and told Shane I would come by his office so we could walk over.
My guest lecture went off very well with the students asking plenty of interesting questions. Then I had lunch with Zoe Chance who is doing some very interesting work with leading companies, applying ideas from behavioral economics. After a couple more meetings, I went to see Shane and we walked over early knowing there would be a big crowd. And we were glad we did, as the auditorium was overflowing by the time the lecture started.
Daniel Kahneman (Danny to his friends) was introduced by another notable person from Yale, Professor Robert Shiller (yes, he of the Case-Shiller Index you may have heard about during the housing crisis). Shiller talked about the widespread impact of Kahneman’s work, especially after the publication of his best seller Thinking, Fast & Slow. Trying to find Kahneman’s connections to Yale, Shiller pointed out that two of his coauthors (Shane Frederick and Nathan Novemsky, both in the marketing department) were at Yale.
And then it was time for Kahneman to speak. His humility, thoughtfulness, and eloquence came through pretty much from the first few words. He started by saying that he doesn’t do university speeches anymore since he is not actively doing any research (he is retired), but could not say no to Bob Shiller. Most of his recent speeches have been about his book, and there had been so many that as a consequence he seems to have forgotten everything else he ever did (laughter!). And that, he said, makes sense because as he points out in the book, we like things that are familiar (more laughter!)....
A friend of mine posted on Facebook that she’d taken a web quiz to tell her which presidential candidate best lined up with her stand on the issues. She was outraged that the web site thought she would vote the way it did. I’m not surprised (by the outrage, not her choice)…it is a case of a badly applied choice technique.
Basically the quiz worked by asking a series of questions to see where she stood on the issues. It then aligns her choices against the stand taken by the candidate (if you want to try one, here is one from the GOP Primaries this year). In essence it is a Configurator. Instead of building the perfect product for you (as you would with a Configurator) you build the perfect candidate. There are a couple of problems with this application.
First, Configurators allow you to build the ideal but generally don’t give a clear idea of what choices you might make if that ideal were not available (our proprietary Texo™ helps overcome that issue). In politics it is not unusual for voting decisions to hinge on a single issue and unlike products you can’t decide to add or subtract an important feature.
Turns out, employers are doing a lot to conserve and protect the environment and natural resources – at least according to TRC’s online panelists we surveyed this spring.
Nearly three-quarters of our panelists who are employed full or part-time told us their employer was actively doing at least one of five activities related to conservation and energy preservation. The larger the employer, the greater the participation. While we can't project our findings to corporate America as a whole, this is certainly encouraging news for our planet.
Shane Frederick (Associate Professor at Yale University’s School of Management) did a talk on Behavioral Economics at our recent research conference that got me thinking. But before we tap into the scary place that is my brain, let’s consider what behavioral economics is. Most of us with a formal business education have taken at least one if not several economics classes, during which we were exposed to market theories based on assumptions that sounded reasonable in principle but that really didn’t represent how things worked in real life. Behavioral economics, Shane started, is the study of economics when those assumptions are relaxed, and the relaxation of one of these assumptions, that people act rationally, is what got my attention.
One of the examples Shane used to make his point involved a pivotal point late in a 2009 football game between the New England Patriots and the Indianapolis Colts. Bill Belichick, the coach of the Patriots, decided to go for it on 4th and 2 deep in his own territory. The attempt failed, the Colts scored after the ensuring change of possession and won the game, and nearly everyone in the sports world pointed to Belichicks' seemingly insane decision. But was it really insane?
Like any research, market research has always recognized that to be certain results of research can be projected to an entire population; you need to eliminate any bias. We worried about things like:
I know this is a simplified view of things, but the above three do get at the major forms of bias that we seek to eliminate in market research. In this blog, I'll focus on representativeness and at some point in the future I'll cover the other two.
Advertisers and researchers do a lot of testing to determine how effective their advertising is prior to launching a campaign or message. We look for ways to get inside consumers’ heads, and as technology improves, we are afforded interesting glimpses into how consumers process information and make decisions. As my colleague Rajan pointed out in his blog different areas of the brain lead to different types of decision-making. Nobel Prize winner Daniel Kahneman posits that human thinking can be classified into two forms, System 1, which operates automatically, and System 2, which requires mental effort (I paraphrase). Jonah Lehrer, author of How We Decide asserts in his blog “Our best decisions are a finely tuned blend of both feeling and reason and the precise mix depends on the situation. When buying a house, for example, it’s best to let our unconscious mull over the many variables. But when we’re picking a stock, intuition often leads us astray. The trick is to determine when to use the different parts of the brain, and to do this, we need to think harder (and smarter) about how we think.”
With all of this exciting work being done in the field of neuroscience and behavioral economics, I wondered what kinds of answers we would get if we simply asked consumers directly what they think motivates them in considering advertising. Do they believe they respond to characters like the Geico gecko? Or is it really just a function of what they need at the time?
In Thinking, Fast & Slow, Nobel winner Daniel Kahneman (click here previous post about Thinking, Fast & Slow) talks about the two selves people have: the experiencing self and the remembering self. The terms are self-explanatory and vacations are a good way to think about them. The part of us that is enjoying the vacation is the experiencing self, while the part that is reliving it later (sometimes years later) is the remembering self. Neither one may be more important, but the emphasis we place on one or the other could determine our behavior. So, for example, you can enjoy the vacation or take plenty of pictures to relive it later, depending on the self that is more important. A way of finding out which self is more important is to ask ourselves whether we would go on a certain vacation if we could only enjoy it, but not take any pictures (or video, etc).
We had a notion here at TRC that by the middle of March most New Year’s Resolutions would have been tossed by the wayside, either in favor of giving up something meaningful for Lent, or the simple acknowledgement that this just isn’t the year to lose 25 pounds. Would folks who made a resolution at the beginning of the year still be keeping that resolution 3 months later?
We kicked around a few hypotheses, and then went about testing them using our online panel of consumers:
So how did our predictions fare?
In his opus Thinking, Fast & Slow, Nobel winner Daniel Kahneman (click here for previous post) relates a story from early in his career when he was leading a team to develop a curriculum and write a textbook on judgment and decision-making in high schools. He had assembled a group of experts and after working diligently for a year they had completed an outline of the syllabus and written two chapters. One fine day when discussing procedures for estimating uncertain quantities, it occurred to him that he should get an estimate from everyone on how long he thought this whole project would take. Being the clever psychologist that he was, rather than ask the group to guess publicly, he asked each person to make a confidential prediction. The mean was about two years and the range was about half a year on either side. In other words, the group was very consistent in its prediction.
Then Kahneman had the idea of asking the curriculum expert in the group, Seymour Fox, for his specific opinion. Only this time he asked Seymour to think about other teams like theirs and asked how long it had taken them to finish. After a long silence the astonishing answer came out. Nearly half the groups never even finished the project. Among those who did the average time taken was about seven years! Seymour Fox also estimated that this group was slightly below average in terms of the skill set it possessed compared to the other groups. The killer, of course, was how long it actually took Kahneman’s group to complete their project. Eight years!
Effectively what had happened was that a group of experts in judgment and decision-making had somehow fooled themselves into thinking way too optimistically about the future and had made predictions based on it. This included the expert who in spite of having the best information somehow ignored that in favor of an optimism bias. As Kahneman graciously adds, it also included a leader who did not pull the plug on a project that would likely take another six years and was a coin toss as to whether it would even be completed.
The biggest lesson Kahneman draws from this episode is that there are two approaches to forecasting which he labels the inside view and the outside view. The inside view is when we focus on the specifics of our own situation, try to form a coherent story and somehow convince ourselves that given the “special” nature of our situation success is just around the corner. In some ways this probably explains the enormously high failure rates of new products and the only slightly lower failure rates of new small businesses. The outside view is one that takes into account the general failure rate of the reference class of objects. Assuming the reference class is properly chosen, the outside view should provide a nice ballpark of where the estimate is going to be. In practice it is better to start there and adjust it using the special knowledge of the inside view and thus avoid embarrassing predictions. Not following this kind of procedure is why we routinely read about say, large transportation projects often running over by years and into several times the original projected cost. It is also why kitchen renovations routinely cost twice the initial estimate for the average household.
So are there specific lessons for market researchers? Of course. One is with the likelihood of success of any kind of new technological advance (mobile, neuro, text analytics, social media monitoring, whatever). Without understanding the reference information for how such new technologies can ultimately fare, we can too easily get caught up in the fanciful nature of a specific technology and make prognostications not just about success, but also about time frames within which such things can come true. On the flip side the death of older technologies can be too gleefully forecast (“Surveys will die in a year!”) because of the glamour of newer techniques if the reference cases are not carefully analyzed....
On a trip to Las Vegas in November 2011 I was twice presented with an option to move to the head of the line – for a price. I could take advantage of “early check-in” by paying $25. And I could get my buffet breakfast right away without waiting in line, again for a small fee. The buffet sign struck me as peculiar, since the 4 people ahead of me didn’t really constitute much of a “line”. I snapped a photo.
The concept of express fees is nothing new – Universal Florida, for example, has offered its ExpressSM Plus Pass for years, affording visitors to skip the regular lines, and as a result experience more attractions during their visit. But the express fee is spreading beyond the domain of the theme park. You can even pay to bypass the long security lines at the airport now, if you’re so inclined.
This got me thinking...who’s in such a rush? And, even more important, who’s willing to fork over some cash so they won’t waste any more time waiting? We put that question to the test with a small web survey among members of TRC’s online panel.
Among the general population of adults, paying for speedy service is a somewhat polarizing notion. While about half of our survey takers are neutral on the concept, 1/3 are pro and 1/5 are anti. We asked about specific situations as well. Paying for early hotel check-in has nearly twice as many fans (23%) as paying for premium seating at a movie (12%) or paying to jump the line at a warehouse store (13%).