Market research and polling is a $10 billion + industry in the US alone. It employs some of the sharpest statisticians and methodological minds that our universities produce. Yet, what is clear is this is an industry that still makes many high profile mistakes.

Here is something we’ve gleaned over the years: When you ask the wrong question on a survey or forget to ask a question, you actually have a reasonable chance to recover from the mistake, provided you notice it. You can change your interpretation of the result based on your imperfect question, you can look to other questions you asked to see if they shed light on the result, and you can look for a secondary source for the information. Not an ideal situation for sure, but often asking a poor question can be a recoverable error. We’ve yet to see the perfect questionnaire or even the perfect question, so in some sense we deal with this issue on every project.

However, if you ask the question of the wrong people, there is usually nothing you can do to correct the error, short of repeating the study. Failure to spot this type of mistake can lead to poor guidance to clients. Posing questions to the wrong audience might well be the most disastrous thing market researchers can do.

And it happens. Early in my career, I remember our firm conducting a phone study where an improper randomization technique occurred which caused us to call one half of a town and not the other. We literally missed everyone living on the “other side of the tracks” who had different opinions of the issues we were covering. The only solution was to repeat the entire project.

More common than actual sample coverage errors is failing to interview the correct individual. This is an ongoing challenge for business-to-business studies. Clients want to talk to a “decision maker,” yet often the decision maker and the user of the product are not the same individual. So, studies end up gathering detailed information on how to improve a product from people who don’t actually use it.

We conduct quite a bit of research with youth audiences. It is common for our clients to ask us to conduct what we call “hearsay” studies. They will ask us, for example, to interview parents and ask them what their kids think about an issue.  Why do that when we can ask the child directly?  We should always be asking questions that our respondents are screened to be the best individuals to answer.

We often see that seasoned researchers or astute up-and-comers realize the importance of sampling and screening.  Which is why, upfront, they dwell more on the sampling aspects of a project than on questionnaire topics and wording.  We always know we are dealing with a client who knows what they are doing if they concentrate the initial conversations on who we want to reach and how we want to reach them, and wait to discuss what we specifically want to ask.