Like any research, market research has always recognized that to be certain results of research can be projected to an entire population; you need to eliminate any bias. We worried about things like:
I know this is a simplified view of things, but the above three do get at the major forms of bias that we seek to eliminate in market research. In this blog, I'll focus on representativeness and at some point in the future I'll cover the other two.
When it comes to response rates, I’ve written many times about this so I’ll be brief. We no longer have the luxury of truly representative samples. Whether a survey is done by phone, web or mail response rates are so low that any reasonable person would not suggest they are representative of anything more than the population of people willing to do surveys.
Phone surveys are probably more representative than web, but they are so fundamentally unrepresentative (refusals to participate, cell only households and call screening mean abysmal response rates) that any difference hardly justifies the time and expense necessary to do them. The only justification, in my view, is when the market is either extremely low incidence or so small geographically that web panels can’t supply enough respondents. Even then I’d first look at alternatives from list sample for the phones to river sampling on the web. I would also add that for projects aimed at older audiences the web might not always be the best choice (though increasingly it is) and of course that in some countries the web is not a viable option.
As I see it we have two choices. We can either raise response rates or learn to live with them. The former is something we as an industry have been talking about since response rates dropped below 60% with virtually no impact. I am very skeptical that we will have any ability to reverse the trend toward lower response rates. Even if we do manage to stem the bleeding or see a slight uptick, we’ll still be looking at response rates that would have been laughed at as late as 20 years ago.
So, our only option in my view is to learn to live with them. It is hard to imagine how much farther advanced new product development research or discrete choice might be if our industry took all the energy wasted on fixing response rates and focused it there
Rich brings a passion for quantitative data and the use of choice to understand consumer behavior to his blog entries. His unique perspective has allowed him to muse on subjects as far afield as Dinosaurs and advanced technology with insight into what each can teach us about doing better research.