I recently had the pleasure of attending a talk by Vicki Morwitz, Professor of Marketing at NYU's Stern School of Business. Vicki spends a lot of time trying to understand how the mere process of surveying people can lead to changes in their behavior - sometimes for an organization's good; sometimes not. She spoke at TRC's Frontiers of Research conference, and as part of her presentation she showed the audience data from an exercise on fruit grouping (or, if you prefer, the grouping of fruit).
Turns out that people who are first exposed to questions with very detailed answer options (e.g., given 9 different colors with which to describe their eyes) will go on to create more narrowly focused fruit categories. In contrast folks primed with more broadly constructed answer categories (e.g., given only 4 different colors) build fewer categories.
Her purpose - to demonstrate how questions asked early in a survey can affect responses later in the survey in a way that can change results. A (perhaps) unintended consequence - getting me to take stock of my role as a research practitioner.
My first reaction: Neat. I should be aware of this effect when designing my questionnaires, because it has potentially big implications for how people evaluate concepts and express their needs.
My second reaction: #!#%&@ - one more "effect" to worry about when I write a questionnaire. And another one that's not necessarily good or bad. I'm exhausted already.
The more you know about consumer behavior, it seems, the harder it is to feel like you can write a "good" questionnaire. That's why I'm glad to have eventually found my way to...
My third reaction: We're in a conversation with the consumer. Knowing that's the most important thing.
Think about the last good conversation you had. You probably enjoyed yourself and felt happy to have engaged in the give and take. Chances are you learned something useful. Without a doubt, you influenced (biased?) your conversation partner through your words, your tone, and your actions, and she did the same. Such influences are unavoidable in conversations, so it's not much of a leap to say they're unavoidable in surveys.
Stack up all the possible opportunities for bias that the typical survey presents, in fact, and you'll hurt yourself trying to see over the top. Instead, think in advance about what you want to get out of your conversation with the consumer, and then design your survey knowing that certain influences and biases will be an inevitable (and even desirable) part of achieving that end.
What kind of conversations are you having with consumers? What kind do you want to have?
For sure it's our job as researchers to keep investigating potential sources of bias, and to help clients avoid misinterpreting results. Our most important mission, however, is to help marketers have better and more productive conversations with consumers. Scrupulously trying to avoid biases won't make that happen. Encouraging them to think about the type of conversations they want to engage in will.