Please don’t judge me for this, but I’ve watched at least half a dozen episodes of America’s Got Talent this summer. It is easy viewing with a variety of acts from daredevils to singing and dancing, and features celebrity judges adding sarcastic asides. But what struck me is how the show’s format points to the essential weakness of rating scales and the strength of choice questions.
In the early “audition” shows, acts come on and perform for a few minutes. The judges then critique them and ultimately vote “yes” or “no”. If two judges vote “no” the act is done. Otherwise the contestants go to Las Vegas for the next round. Now while “yes” or “no” is in fact a choice, it is really nothing more than a disguised rating. The reason is there is no constraint. They don’t have a limit on how many people go forward. This is like reading a list of features and asking respondents which ones are important to them (anyone who has done market research knows the answer to such questions is generally “everything is important”).
Once in Vegas the hard work begins. This season about 120 acts made it there, but only 60 are needed for the competition. So the judges had to decide which 60 would get to the next stage. To do this they picked 30 acts that they thought were good enough to go on and 60 that they wanted to see again to pick the other 30. The remaining 30 were called in and summarily told that they were done (so yes, they flew them to Vegas just to tell them this). Frankly I’d been surprised by many of the acts that got to go to Vegas, so I wasn’t surprised by the choices.
The key here was that unlike the early rounds…they now had a constraint. As with Max Diff (where you have to pick winners and losers) and Conjoint (where you are constrained by the mix of features and levels), they now had to make real choices. In this case, many were not hard (though telling 10 year olds they are done can’t be easy…even if they clearly are not good enough). The 60 remaining acts were not all great (many were not even good in my opinion), but they were far better than the 60 sent packing.
From here the tournament becomes more like our proprietary Bracket™ technique. Performances are compared to each other with some getting to move on (and perform against other winning acts) and some being done. In the end only one act will win…the one that is most popular among the dedicated fans of the show. This is exactly how good market research should work…force hard choices to drive the best product, message, segmentation solution or price using pricing research....
A friend of mine posted on Facebook that she’d taken a web quiz to tell her which presidential candidate best lined up with her stand on the issues. She was outraged that the web site thought she would vote the way it did. I’m not surprised (by the outrage, not her choice)…it is a case of a badly applied choice technique.
Basically the quiz worked by asking a series of questions to see where she stood on the issues. It then aligns her choices against the stand taken by the candidate (if you want to try one, here is one from the GOP Primaries this year). In essence it is a Configurator. Instead of building the perfect product for you (as you would with a Configurator) you build the perfect candidate. There are a couple of problems with this application.
First, Configurators allow you to build the ideal but generally don’t give a clear idea of what choices you might make if that ideal were not available (our proprietary Texo™ helps overcome that issue). In politics it is not unusual for voting decisions to hinge on a single issue and unlike products you can’t decide to add or subtract an important feature.
In Thinking, Fast & Slow, Nobel winner Daniel Kahneman (click here previous post about Thinking, Fast & Slow) talks about the two selves people have: the experiencing self and the remembering self. The terms are self-explanatory and vacations are a good way to think about them. The part of us that is enjoying the vacation is the experiencing self, while the part that is reliving it later (sometimes years later) is the remembering self. Neither one may be more important, but the emphasis we place on one or the other could determine our behavior. So, for example, you can enjoy the vacation or take plenty of pictures to relive it later, depending on the self that is more important. A way of finding out which self is more important is to ask ourselves whether we would go on a certain vacation if we could only enjoy it, but not take any pictures (or video, etc).
A former colleague of mine used to tell us to “torture the data until it confessed”. In other words, don’t just stop your investigation at the first finding. But rather, keep poking, prodding, flipping and coercing until you feel you’ve uncovered all the data has to give. Ah…images of Jack Bauer doing his thing flash through my mind just thinking about our own data “torture” sessions.
All kidding aside, what my colleague was really trying to say was spot on. I’m sure we’ve all known researchers who habitually stop at the first find. They rarely take the time to consider different ways of looking at data, of considering the message within.
My seven year old son gets a $2 per week allowance. He doesn't really do anything to earn this money. Rather I give him (and his brother) an allowance to teach them how to save for things that they want. Implied, and in fact part of the bargain, is that they can't hassle me for Pokémon cards, or Wii games, or anything else they "need", because they have their own money. Well, about a month or two ago my seven year old mandated that I start paying him with a $2 bill. Yikes! Where was I going to get even one $2 bill, let alone one every week?
As we consider my situation, let's juxtapose something we've all been hearing for 10 plus years now. The brick and mortar (fill in the blank) is antiquated, and on its way to irrelevance. The Internet is the way that EVERYBODY is going to shop for and do EVERYTHING! Heck, I've heard it so many times and for so long that I agree with it, which is odd since the only items I consistently buy online are books, DVDs and music.
A few weeks back we decided to get a new TV. As a researcher I started doing my due diligence checking out a variety of sources. Our old TV had been around since the early days of HD and was a CRT to boot (though still HD - c'mon what do you think I am!). So I was really looking forward to buying something that weighed considerably less, took up a lot less space and looked a lot cooler. It was fun going through the various attributes - 760/1080, LCD/Plasma, LED backlighting, HDMI connections, Internet Apps, 3D! It's been a while since my engineering days when I studied the innards of TVs, and the technology has certainly evolved remarkably since then.
I narrowed my choices down to a few brands primarily based on the screen size that would fit the space, and really started focusing on a Vizio model. As you may know, Vizio has become a real player over the last few years. Initially they competed on price, but more recently have made a splash with features and quality. Some of the reviews I saw online in fact, put Vizio near the top of the rankings. I was particularly taken by the Vizio Internet Apps (VIA) which makes it absurdly easy to get internet content directly on your TV. I was really looking forward to the Netflix app. Not being a current subscriber, the ability to stream so many movies simply and directly through my TV was huge. The price was very competitive too, so we decided to go with Vizio.
The purchase was made through Costco online and the TV arrived about a week later. After upgrading some accessories, I started connecting it. The process was very easy. The remote was simple and elegantly designed. Best of all it had a slide out QWERTY keyboard (a big hit with the kids) to access Facebook, Twitter, etc. The onscreen instructions for set-up were very easy to follow and the whole thing took very little time. Then we sat back to enjoy the content and that's when the trouble started.
The picture started shaking, froze and blanked out. Just for a few seconds. That's normal when connecting new devices, I assured everyone confidently. Then it happened again. And again. Uh oh! When it didn't do any of this, the HD somehow became SD. I was getting very concerned, so I called Vizio's customer service. The rep tried a few things but nothing changed. He said they could send a repair person over "in 5 to 7 business days". Meanwhile I had unhappy kids at home and can't watch the Phillies or the NBA Finals. That did it. Off we went TV shopping again, but had decided to eliminate Vizio from our consideration set.
The rational part of me said the glitch in that one TV was very unlikely to be repeated again. Vizio has been tested by expert reviewers and gets high marks. The predictably irrational part said I would be a fool to buy the same TV again. Guess who won? So this time I bought a Panasonic from Best Buy after grilling the sales associate with endless questions. The TV worked beautifully right out of the box. It has the Internet apps I coveted, but the Vizio certainly had more features and flexibility. But no matter....
I recently had the pleasure of attending a talk by Vicki Morwitz, Professor of Marketing at NYU's Stern School of Business. Vicki spends a lot of time trying to understand how the mere process of surveying people can lead to changes in their behavior - sometimes for an organization's good; sometimes not. She spoke at TRC's Frontiers of Research conference, and as part of her presentation she showed the audience data from an exercise on fruit grouping (or, if you prefer, the grouping of fruit).
Turns out that people who are first exposed to questions with very detailed answer options (e.g., given 9 different colors with which to describe their eyes) will go on to create more narrowly focused fruit categories. In contrast folks primed with more broadly constructed answer categories (e.g., given only 4 different colors) build fewer categories.
Her purpose - to demonstrate how questions asked early in a survey can affect responses later in the survey in a way that can change results. A (perhaps) unintended consequence - getting me to take stock of my role as a research practitioner.
In his book on the neuroscience of decision making, How We Decide, Jonah Lehrer talks about the case of a patient who has damaged a part of his brain (specifically the orbitofrontal cortex) and is hence terminally unable to make any decisions. Every single decision, no matter how trivial, seems complicated to the point where it cannot be made. This is generally not a problem for most normal people, right? In fact, the accepted wisdom is that people are quite good at taking somewhat complicated decisions and simplifying them (in many cases rather efficiently) and moving on with their life.
Now there is new research from our friend Oded Netzer at Columbia and his colleagues Rom Schrift and Ran Kivetz that shows that not only do people simplify, but sometimes they also complicate the decision-making process unnecessarily. They studied this through a variety of experiments, many designed to rule out competing explanations. Let's talk about one of those to understand what they did.
How do you make choices in your life? Even simple ones like chocolate cake or fruit salad for a snack? Are you completely rational about the process, calculating the costs and benefits properly before choosing (also known as the cognitive approach)? Or are you more likely to go by feel, allowing your emotions to guide the choice (the affective approach)? Traditionally, researchers have favored the rational model, but more recently the emotional side has been getting more attention. Regular folks may even argue that they use both approaches depending on the situation, even though they may not know which one predominates without their knowledge. But can your decision-making process, and thus the choices you make, be influenced by external conditions to the extent that you will switch from one mode to another? That was the question that drove two researchers in their quest to understand the process of making choices.