Winners vs. Losers, Bracket™ vs. Max-Diff
If someone asked you to put 70 items in order, from best to worst, how would you go about doing that?
That was the question we had before us when we wanted to figure out how to get TV viewers to tell us their preferences for scripted TV shows. Our goal was to determine whether the TV shows that TV critics rank as the "best" are those that TV viewers would also rank as best. And to do that, we needed TV viewers to prioritize 70 TV shows.
We considered using a Max-Diff (or Maximum Difference) analysis. Max-Diff is a choice method in which respondents are shown sets of items and asked to pick the best and worst in each set. The number of items shown per set usually varies from three to five. The composition and order of the sets are pre-assigned. In the end, respondent-level scores are produced for all of the items evaluated.
Max-Diff is an appropriate and robust technique that produces preference scores with greater discrimination than other approaches, such as rating scales. However, when there are a large number of items to prioritize, Max-Diff can be very tedious for a respondent.
For the TV show project, we opted to employ Bracket™ instead. Bracket™ differs from Max-Diff in that it is conducted in "rounds;" the losers from the initial rounds are not shown again in later rounds – only the winners advance. In subsequent rounds the decisions are tougher – but they're also more relevant to the respondent.
The key advantages of Bracket™ over Max-Diff are that it allows for a large number of items to be rated easily and can do so with fewer "choices," making it far less tedious. Despite having fewer choices, and thus collecting less information, in side-by-side tests Bracket™ produces high-quality results, quite similar to Max-Diff.
Respondents were asked to complete 25 tasks in our 70-item Bracket™. Had this been designed as a Max-Diff, the respondents would have had to complete 42 tasks, a far more exhausting exercise, and one which we would have been unlikely to implement.
In the end, we were able to establish a rank ordering of the 70 shows, which showed clear "winners" at the top of the list. NCIS (CBS), The Big Bang Theory (CBS) and The Walking Dead (AMC) were the top 3, with NCIS the stand-out. Family Guy (Fox) and Game of Thrones (HBO) were in the next tier. Using this technique, a few items floated to the top of a very long list – showing discrimination among many items can still be achieved as long as the right approach or tool is utilized.
For additional details on the results, click here.