Welcome visitor you can log in or create an account


Consumer Insights. Market Innovation.

Posted by on in A Day in a (MR) Life
  • Font size: Larger Smaller
  • Hits: 369441
  • Print

Knowing Enough to Make the Right Decision

knowing enough to make the right decisionSometimes as researchers we get too hung up on knowing everything.   We get frustrated by interesting findings that can't be explained with the available data and this can cause us to miss important insights. I suspect that the proliferation of available data will do little to help fill in the blanks...in fact, it might make the problem worse. A simple exercise in text analytics highlights this point.

There are now an array of tools available to help quantify and understand massive amounts of text.  For example,  at one of our conferences last year, Oded Netzer of Columbia University presented an amazing tool that analyses message boards and other online forums to learn about specific markets (slides can be found at:  http://www.trchome.com/research-knowledge/conferences/437). Tools like these provide a rich and valuable source of data, but insight can also be gleaned from far more simple approaches.

Aaron Gerow (Trinity College) and Mark Keane (University College Dublin) conducted a simple, yet insightful analysis of four years worth of articles in three publications that dealt with the stock market. Unlike many text analytic exercises, they didn’t attempt to figure out what was being said. Instead they simply counted up how often various words were being used over time. This provided some insight into the diversity of opinions being shared…namely, if they found less different words being used (but on average with more frequency) it probably indicated that opinions were more similar than if a greater number of different words were being used.

They then plotted these changes in diversity against stock market performance and found something very interesting. As markets went up, the diversity of opinion went down. On the other hand, as markets dropped, diversity went up. They can’t explain why this is the case. It may be that there are only a few explanations for a good market and many for a bad one.

Another interesting point is that both the steady drop in the market (that started in 2007) and the recovery (that started in 2009) were preceded by changes in the diversity of the words being used by financial reporters. The data can’t tell us if this change was brought about by reporters sensing the market direction was about to change (and that in turn changing their writing) or if the change in their writing caused the change in market direction.

While both the reason for the changes in diversity of words used and understanding which was the cause and which was the effect are interesting questions, they are not essential to making decisions on market timing. As an investor what really matters is that you’d be wise to be wary when you note financial reporters start to have more diverse opinions and be ready to invest when those opinions start to coalesce again.

In other words, sometimes we don’t need all the answers as long as you have the most useful ones.

Finally, it is worth noting that this is based essentially on two data points and that my record as an amateur financial advisor is far worse than my record as a professional researcher so probably best not to risk your nest egg on it.

Tagged in: Statistics Text Mining

President, TRC

Rich brings a passion for quantitative data and the use of choice to understand consumer behavior to his blog entries. His unique perspective has allowed him to muse on subjects as far afield as Dinosaurs and advanced technology with insight into what each can teach us about doing better research.  


  • No comments made yet. Be the first to submit a comment

Leave your comment

Guest Friday, 18 September 2020

Want to know more?

Give us a few details so we can discuss possible solutions.

Please provide your Name.
Please provide a valid Email.
Please provide your Phone.
Please provide your Comments.
Invalid Input
Our Phone Number is 1-800-275-2827
 Find TRC on facebook  Follow us on twitter  Find TRC on LinkedIn

Our Clients