Please don’t judge me for this, but I’ve watched at least half a dozen episodes of America’s Got Talent this summer. It is easy viewing with a variety of acts from daredevils to singing and dancing, and features celebrity judges adding sarcastic asides. But what struck me is how the show’s format points to the essential weakness of rating scales and the strength of choice questions.
In the early “audition” shows, acts come on and perform for a few minutes. The judges then critique them and ultimately vote “yes” or “no”. If two judges vote “no” the act is done. Otherwise the contestants go to Las Vegas for the next round. Now while “yes” or “no” is in fact a choice, it is really nothing more than a disguised rating. The reason is there is no constraint. They don’t have a limit on how many people go forward. This is like reading a list of features and asking respondents which ones are important to them (anyone who has done market research knows the answer to such questions is generally “everything is important”).
Once in Vegas the hard work begins. This season about 120 acts made it there, but only 60 are needed for the competition. So the judges had to decide which 60 would get to the next stage. To do this they picked 30 acts that they thought were good enough to go on and 60 that they wanted to see again to pick the other 30. The remaining 30 were called in and summarily told that they were done (so yes, they flew them to Vegas just to tell them this). Frankly I’d been surprised by many of the acts that got to go to Vegas, so I wasn’t surprised by the choices.
The key here was that unlike the early rounds…they now had a constraint. As with Max Diff (where you have to pick winners and losers) and Conjoint (where you are constrained by the mix of features and levels), they now had to make real choices. In this case, many were not hard (though telling 10 year olds they are done can’t be easy…even if they clearly are not good enough). The 60 remaining acts were not all great (many were not even good in my opinion), but they were far better than the 60 sent packing.
From here the tournament becomes more like our proprietary Bracket™ technique. Performances are compared to each other with some getting to move on (and perform against other winning acts) and some being done. In the end only one act will win…the one that is most popular among the dedicated fans of the show. This is exactly how good market research should work…force hard choices to drive the best product, message, segmentation solution or price using pricing research....