Welcome visitor you can log in or create an account


Consumer Insights. Market Innovation.

Posted by on in Market Research
  • Font size: Larger Smaller
  • Hits: 7754
  • Print

My Results Are Biased and I Don't Care!

Like any research, market research has always recognized that to be certain results of research can be projected to an entire population; you need to eliminate any bias. We worried about things like:

  • Representativeness Effects – Needed to not only make sure we selected a random representative sample, but then do everything possible to maximize the percentage of people who completed the survey.
  • Interviewer Effects – Surveys needed to be done identically.   If one was done by mail, all should be with identical forms. If done by phone interviewers needed to be careful not to lead respondents and to keep pacing at consistent rate.
  • Framing Effects– If responses from one question are going to potentially bias a future response then the order should be changed to reflect it. In cases where changing the order merely changes which question biases which, use rotation or split samples so that bias effects can be measured and softened.

I know this is a simplified view of things, but the above three do get at the major forms of bias that we seek to eliminate in market research. In this blog, I'll focus on representativeness and at some point in the future I'll cover the other two.

When it comes to response rates, I’ve written many times about this so I’ll be brief.   We no longer have the luxury of truly representative samples. Whether a survey is done by phone, web or mail response rates are so low that any reasonable person would not suggest they are representative of anything more than the population of people willing to do surveys.  

Phone surveys are probably more representative than web, but they are so fundamentally unrepresentative (refusals to participate, cell only households and call screening mean abysmal response rates) that any difference hardly justifies the time and expense necessary to do them.   The only justification, in my view, is when the market is either extremely low incidence or so small geographically that web panels can’t supply enough respondents.   Even then I’d first look at alternatives from list sample for the phones to river sampling on the web.   I would also add that for projects aimed at older audiences the web might not always be the best choice (though increasingly it is) and of course that in some countries the web is not a viable option.

As I see it we have two choices. We can either raise response rates or learn to live with them.   The former is something we as an industry have been talking about since response rates dropped below 60% with virtually no impact.     I am very skeptical that we will have any ability to reverse the trend toward lower response rates.   Even if we do manage to stem the bleeding or see a slight uptick, we’ll still be looking at response rates that would have been laughed at as late as 20 years ago.

So, our only option in my view is to learn to live with them.   It is hard to imagine how much farther advanced new product development research or discrete choice might be if our industry took all the energy wasted on fixing response rates and focused it there

President, TRC

Rich brings a passion for quantitative data and the use of choice to understand consumer behavior to his blog entries. His unique perspective has allowed him to muse on subjects as far afield as Dinosaurs and advanced technology with insight into what each can teach us about doing better research.  


  • No comments made yet. Be the first to submit a comment

Leave your comment

Guest Tuesday, 22 September 2020

Want to know more?

Give us a few details so we can discuss possible solutions.

Please provide your Name.
Please provide a valid Email.
Please provide your Phone.
Please provide your Comments.
Invalid Input
Our Phone Number is 1-800-275-2827
 Find TRC on facebook  Follow us on twitter  Find TRC on LinkedIn

Our Clients