My daughter was performing in The Music Man this summer and after seeing the show a number of times, I realized it speaks to the perils of poor planning…in forming a boys band and in conducting complex research.  

For those of you who have not seen it, the show is about a con artist who gets a town to buy instruments and uniforms for a boys band in exchange for which he promises he’ll teach them all how to play. When they discover he is a fraud they threaten to tar and feather him, but (spoiler alert) his girl friend gets the boys together to march into town and play. Despite the fact that they are awful, the parents can’t help but be proud and everyone lives happily ever after.

It is to some extent another example of how good we are at rationalizing. The parents wanted the band to be good and so they convinced themselves that they were. The same thing can happen with research…everyone wants to believe the results so they do…even when perhaps they should not.

I’ve spent my career talking about how important it is to know where your data have been. Bias introduced by poor interviewers, poorly written scripts, unrepresentative sample and so on will impact results AND yet these flawed data will still produce cross tabs and analytics. Rarely will they be so far off that the results can be dismissed out of hand.

The problem only gets worse when using advanced methods. A poorly designed conjoint will still produce results. Again, more often than not these results will be such that the great rationalization ability of humans will make them seem reasonable.

So, don’t get caught up in this trap. Take the time up front to consider the best design for solving your research challenge. Use your past experience to find the best solution. For example, at TRC we have hundreds of conjoint studies under our belt, from simple discrete choice designs to very complicated adaptive conjoint studies…these inform on what works and what doesn’t. Understanding in advance how things like uneven numbers of levels for different features will impact utility scores helps us (and our clients) avoid the rationalization trap.

Also, in fairness to my daughter, I can say that I have plenty of independent validation of her talent so I’m not like the parents of River City…I know I should be proud. Perhaps the only rationalization I’m doing is to convince myself that I’m writing a research blog when I’m really just bragging….