TRC

Ever wondered how paleontologists know what a dinosaur weighs? OK, me neither, but an article I read in The Economist points to mistakes in past methods and I believe understanding these mistakes can teach us a lot about how to be better researchers.

A dinosaur’s weight is estimated by taking the bone structure and weight of existing animals and then through linear regression predicting the weight of dinosaurs using only their bone structures. For example, a Brontosaurus (technically called an Apatosaurus, but I learned my dinosaur names watching The Flintstones) is estimated to weigh about as much as seven African elephants.

Dr. Gary Packard of Colorado State University wondered how well these equations would do at predicting the weight of living animals. In essence, he pretended we don’t know how much an elephant weighs. He took the weight and bone structure of smaller animals and then used a linear regression to predict an elephant’s weight using only its bone structure. The result was 50% more than an elephant weighs.

Dr. Packard then created a nonlinear regression model which far more accurately predicted the weight of large animals based only on their bone structure. This same formula would predict that a Brontosaurus would weigh the equivalent of only three African elephants…almost four elephants less than was previously estimated.

While Paleontology and market research are clearly unrelated fields, this example does point out some things that we, as researchers, should keep in mind so we can avoid our own “four elephant” mistakes.

Just as Paleontologists had to extrapolate, we often have to extrapolate. Whether that means stretching a conjoint simulator to the limit or predicting take rates at price points beyond those tested, we need to be as vigilant as Dr. Packard to ensure that the answers are as accurate as they possibly can be. For example, he essentially used the elephant as a hold out sample to test his results. We should be making more frequent and better use of hold out samples to separate good results from data mining run amuck.

We often evaluate data using linear regressions…the same linear regressions that overstated dinosaurs’ weight. For this to be correct, we have to assume that the relationship between the attributes rated and say brand perception is linear. So let’s say an airline has a great safety record and they give passengers fresh baked cookies during the flight. A linear regression would assume that an airline’s safety record and the cookies could have positive and a negative impact on brand perception. I don’t think that makes logical sense. Most people have an expectation that the airline will get us home safely so they won’t reward an airline for doing so...though they will punish an airline when the perception of safety is lost. Conversely, most people probably don’t think worse of an airline that doesn’t provide cookies, though they’d think better of one that does. In short, both of these relationships are probably asymmetrical. As with weighing dinosaurs we need to use a non-linear method to truly understand the data

Linear thinking would also have us assume that the improvement in customer experience necessary to move someone’s perception from a 5 to 6 is identical to what it takes to move from a 9 to a 10. Does that make any sense? Dr. Vikas Mittal of Rice University recently compared the two and showed clear differences with regards to actual repurchase behavior as compared to stated intent.

Finally, we should always put our data in context so that those applying it know how much confidence to put in it. For all of his vigilance Dr. Packard’s methods may not be more accurate than those used in the past. What if the weight structure of Dinosaurs is completely different than any animals alive today? Until Jurassic Park opens, we won’t really know for sure.

We’d all do well to keep these lessons in mind. If we do, and you had to know this was coming, then market researchers can avoid the dinosaur’s fate.

Comments (0)

Subscribe to this comment's feed

Name

Email

Website

Title

Comment

smaller | bigger

Subscribe via email (registered users only)

I have read and agree to the Terms of Usage.

Add Comment