Archive for the 'Insights Technology' Category

What is the Potential for Chatbots for Market Research?

AI tools, specifically chatbots such as Chat GBT, are buzzing among market researchers. While chatbots aren’t brand new, Chat GBT’s capabilities have brought the technology to the minds of many for the first time. Chat GBT seems uncannily accurate and less “artificial” than its AI predecessors. It is as if we are watching Frankenstein’s monster become aware in real time.

History shows that there is always a hype curve with new technology. Initially, there is strong enthusiasm that forecasts a greater potential than will ever be realized. Then, as reality sets in, people adjust their expectations downward – usually too far- and disillusionment with the technology ensues. Finally, the technology finds its level – usually below its initial expectations and greater than its disillusionment level.

This is an MBA-like way of saying that chatbots probably won’t take over our field, but they can potentially change how researchers work. They can create efficiencies by saving time and money and may launch new ways of researching consumers that have not been undertaken before.

Chatbots have the potential to significantly improve market research by providing a fast and efficient way to collect data from a large number of people in a short amount of time. Chatbots can be programmed to ask specific questions and record responses in real-time, making the data collection process more efficient and accurate.

Some potential benefits of using chatbots for market research include:

  • Improved efficiency: Chatbots can collect data from multiple respondents simultaneously, reducing the time and resources required for data collection.
  • Increased accuracy: Chatbots can ask questions in a consistent and unbiased manner, reducing the risk of human error and bias in data collection.
  • Greater reach: Chatbots can reach a larger audience, including those who may not typically participate in market research studies.
  • Real-time insights: Chatbots can provide real-time insights into consumer preferences and behaviors, allowing businesses to quickly respond to changes in the market.
  • Cost-effectiveness: Chatbots can be programmed to automate the data collection process, reducing the need for human researchers and lowering the overall cost of market research.

I see immediate potential for Chat GBT in qualitative research. Question probing can be more instantaneous, automatic, and patient for online bulletin boards than a human interviewer can provide. I can envision automated IDIs being developed, which will be online one-on-one conversations that do not require an interviewer. This will make IDIs more affordable, and more will be done. This is a good thing, as quantitative studies are suffering from data quality issues and IDIs are becoming a more critical element of many research programs.

I see less of an immediate potential for open-ends in quantitative surveys. Yes, chatbots can probe an open-ended response. But, their ability to do this well requires a good initial response from the respondent. The quality of open-ended responses in online surveys tends to be weak and is declining. The bot requires a thoughtful response to probe on, so I don’t see this as a good use of these bots until we improve the initial responses we get. For now, I see this being much more effective when surveying an engaged audience, such as customers, or in business-to-business studies.

At some point, I suspect that chatbots will make quantitative surveys more adaptive. Follow-up survey questions can automatically appear depending on earlier responses. Yes, we can do that now with survey logic and branching, but the bots may be able to ask questions we haven’t even thought of. When analyzing data at the end of every research project, we always discover questions we wish we had asked. Chatbots may be able to develop those for us before we even see the data.

On the downside, chatbots will also beget even more survey fraud. To date, it is pretty easy to look at open-end responses and tell which ones were written by a bot. That may not be as easy in the future. More fraudsters will defeat our defenses. This begs a strange question: will we use AI to detect the presence of AI fraud tools?

It is essential to know how chatbots work. They aren’t human. But, like humans, they are “trained” on past information. A chatbot’s advantage over humans is it can be trained on a vast amount of data, which gives them enormous potential to “think” faster than a human can.

But, because they are trained on past information, chatbots will be most accurate when tasked with researching issues with a past analog. They are likely less accurate in predicting the future for new and novel products and questions. And that is often the most important role for research. The job of market research is to provide information to guide future decisions.

As currently designed, these bots will likely be more accurate and helpful in areas where technical, encyclopedic knowledge prevails. We have heard how Chat GBT can ace the SAT exam. But, at least currently, it does much better on AP exams in Science than in English.

Back to the original point: chatbots are probably over-hyped right now, but don’t discount their potential too much. What if I told you that two of the paragraphs of this post were generated by Chat GBT? Do you think you can tell which ones?

Be afraid. Be very afraid!

The Insight that Insights Technology is Missing

The market research insights industry has long been characterized by a resistance to change. This likely results from the academic nature of what we do. We don’t like to adopt new ways of doing things until they have been proven and studied.

I would posit that the insights industry has not seen much change since the transition from telephone to online research occurred in the early 2000s. And even that transition created discord within the industry, with many traditional firms resistant to moving on from telephone studies because online data collection had not been thoroughly studied and vetted.

In the past few years, the insights industry has seen an influx of capital, mostly from private equity and venture capital firms. The conditions for this cash infusion have been ripe: a strong and growing demand for insights, a conservative industry that is slow to adapt, and new technologies arising that automate many parts of a research project have all come together simultaneously.

Investing organizations see this enormous business opportunity. Research revenues are growing, and new technologies are lowering costs and shortening project timeframes. It is a combustible business situation that needs a capital accelerant.

Old school researchers, such as myself, are becoming nervous. We worry that automation will harm our businesses and that the trend toward DIY projects will result in poor-quality studies. Technology is threatening the business models under which we operate.

The trends toward investment in automation in the insights industry are clear. Insights professionals need to embrace this and not fight it.

However, although the movement toward automation will result in faster and cheaper studies, this investment ignores the threats that declining data quality creates. In the long run, this automation will accelerate the decline in data quality rather than improve it.

It is great that we are finding ways to automate time-consuming research tasks, such as questionnaire authoring, sampling, weighting, and reporting. This frees up researchers to concentrate on drawing insights out of the data. But, we can apply all the automation in the world to the process, yet if we do not do something about data quality, it will not increase the value clients receive.

I argue in POLL-ARIZED that the elephant in the research room is the fact that very few people want to take our surveys anymore. When I began in this industry, I routinely fielded telephone projects with 70-80% response rates. Currently, telephone and online response rates are between 3-4% for most projects.

Response rates are not everything. You can make a compelling argument that they do not matter at all. There is no problem as long as the 3-4% response we get is representative. I would rather have a representative 3% answer a study than a biased 50%.

But, the fundamental problem is that this 3-4% is not representative. Only about 10% of the US population is currently willing to take surveys. What is happening is that this same 10% is being surveyed repeatedly. In the most recent project Crux fielded, respondents had taken an average of 8 surveys in the past two weeks. So, we have about 10% of the population taking surveys every other day, and our challenge is to make them represent the rest of the population.

Automate all you want, but the data that are the backbone of the insights we are producing quickly and cheaply is of historically low quality.

The new investment flooding into research technology will contribute to this problem. More studies will be done that are poorly designed, with long, tortuous questionnaires. Many more surveys will be conducted, fewer people will be willing to take them, and response rates will continue to fall.

There are plenty of methodologists working on these problems. But, for the most part, they are working on new ways to weight the data we can obtain rather than on ways to compel more response. They are improving data quality, but only slightly, and the insights field continues to ignore the most fundamental problem we have: people do not want to take our surveys.

For the long-term health of our field, that is where the investment should go.

In POLL-ARIZED, I list ten potential solutions to this problem. I am not optimistic that any of them will be able to stem the trend toward poor data quality. But, I am continually frustrated that our industry has not come together to work towards expanding respondent trust and the base of people willing to take part in our projects.

The trend towards research technology and automation is inevitable. It will be profitable. But, unless we address data quality issues, it will ultimately hasten the decline of this field.

Visit the Crux Research Website

Enter your email address to follow this blog and receive notifications of new posts by email.