Some months ago, Lily Allen mistakenly received an email containing harsh test group feedback regarding her new album. Select audience members believed the singer to be retired and threw in some comments that I won’t quote. If you are curious, the link to her Popjustice interview will let you see them in a more raw form. Allen returned the favor with some criticism on market research itself:
“The thing is, people who take part in market research: are they really representative of the marketplace? Probably not.” –Lily Allen
The singer brings up a valid concern. One of the many questions I pondered five months ago when I first took my current researcher-in-training position with TRC. Researchers are responsible for engaging a representative sample and delivering insights. How do we uphold those standards to ensure quality? Now that I have put in some time and have a few projects under my belt, I have assembled a starter list to address those concerns:
In order to complete any research project, there needs to be a clear objective. What are we measuring? Are we using one of our streamlined products, such a Message Test Express™, or will there be a conjoint involved? This may seem obvious, but it is also critical. A team of people is behind each project at TRC; including account executives, research managers, project directors, and various data experts. More importantly, the client should also be on the same page and kept in the loop. Was the artist the main client for the research done? My best guess is no, the feedback given was not meant to be a tool to rework the album.
Was the research done on Lily Allen’s album even meant to be representative? Qualitative interviews can produce deep insights among a small, non-representative, group of people. This can be done as a starting point or a follow-up to a project, or even stand alone, depending on the project objectives....
I read a blurb in The Economist about UFO sightings. They charted some 90,000 reports and found that UFO's are, as they put it, "considerate". They tend not to interrupt the work day or sleep. Rather, they tend to be seen far more often in the evening (peaking around 10PM) and more on Friday nights than other nights.
The Economist dubbed the hours of maximum UFO activity to be "drinking hours" and implied that in fact that drinking was the cause of all those sightings.
As researchers, we know that correlation does not mean causation. Of course their analysis is interesting and possibly correct, but it is superficial. One could argue (and I'm sure certain "experts" on the History Channel would) that it is in fact the UFO activity that causes people to want to drink, but by limiting their analysis to two factors (time of day/number of sightings), The Economist ignore other explanations.
For example, the low number of sightings during sleeping hours would make perfect sense (most of us sleep indoors with our eyes closed). The same might be true for the lower number during work hours (many people don't have ready access to a window and those who do are often focused on their computer screen and not the little green men taking soil samples out the window).
As researchers, we need to consider all the possibilities. Questionnaires should be constructed to include questions that help us understand all the factors that drive decision making. Analysis should, where possible, use multivariate techniques so that we can truly measure the impact of one factor over another. Of course, constructing questions that allow respondents to express their thinking is also key...while a long attribute rating battery might seem like it is being "comprehensive" it is more likely mind numbing for the respondent. We of course prefer to use techniques like Max-Diff, Bracket™ or Discrete Choice to figure out what drives behavior.
Hopefully I've given you something to think about tonight when you are sitting on the porch, having a drink and watching the skies.
Advertisers and researchers do a lot of testing to determine how effective their advertising is prior to launching a campaign or message. We look for ways to get inside consumers’ heads, and as technology improves, we are afforded interesting glimpses into how consumers process information and make decisions. As my colleague Rajan pointed out in his blog different areas of the brain lead to different types of decision-making. Nobel Prize winner Daniel Kahneman posits that human thinking can be classified into two forms, System 1, which operates automatically, and System 2, which requires mental effort (I paraphrase). Jonah Lehrer, author of How We Decide asserts in his blog “Our best decisions are a finely tuned blend of both feeling and reason and the precise mix depends on the situation. When buying a house, for example, it’s best to let our unconscious mull over the many variables. But when we’re picking a stock, intuition often leads us astray. The trick is to determine when to use the different parts of the brain, and to do this, we need to think harder (and smarter) about how we think.”
With all of this exciting work being done in the field of neuroscience and behavioral economics, I wondered what kinds of answers we would get if we simply asked consumers directly what they think motivates them in considering advertising. Do they believe they respond to characters like the Geico gecko? Or is it really just a function of what they need at the time?
A few months ago I posted that we researched 18 factors in deciding which movie to see and where to see it. We reported that “It’s in 3D” was at the bottom of the list, and concluded that 3-D was unlikely to save the American movie box office.
What made the top of the list was “I like the plot or story,” followed by “It is in my favorite movie genre” and “It has my favorite stars.”
But surely the plot isn’t the critical decision-maker for every movie-goer; there must be groups of viewers whose decisions revolve around some of the other items on that list. We took their ratings and ran a segmentation analysis. While this type of analysis is done on a much grander scale by researchers in the movie industry, we thought it would be interesting to do some analysis of our own.