Archive for the 'Uncategorized' Category

Let’s Appreciate Statisticians Who Make Data Understandable

Statistical analyses are amazing, underrated tools. All scientific fields depend on discoveries in statistics to make inferences and draw conclusions. Without statistics, advances in engineering, medicine, and science that have greatly improved the quality of life would not have been possible. Statistics is the Rodney Dangerfield of academic subjects – it never gets the respect it deserves.

Statistics is central to market research and polling. We use statistics to describe our findings and understand the relationships between variables in our data sets. Statistics are the most important tools we have as researchers.

However, we often misuse these tools. I firmly believe that pollsters and market researchers overdo it with statistics. Basic, statistical analyses are easy to understand, but complicated ones are not. Researchers like to get into complex statistics because it lends an air of expertise to what we do.

Unfortunately, most sophisticated techniques are impossible to convey to “normal” people who may not have a statistical background, and this tends to describe the decision-makers we support.

I learned long ago that when working with a dataset, any result that will be meaningful will likely be uncovered by using simple descriptive statistics and cross-tabulations. Multivariate techniques can tease out more subtle relationships in the data. Still, the clients (primarily marketers) we work with are not looking for subtleties – they want some conclusions that leap off the page from the data.

If a result is so subtle that it needs complicated statistics to find, it is likely not a large enough result to be acted upon by a client.

Because of this, we tend to use multivariate techniques to confirm what we see with more straightforward methods. Not always – as there are certainly times when the client objectives call for sophisticated techniques. But, as researchers, our default should be to use the most straightforward designs possible.

I always admire researchers who make complicated things understandable. That should be the goal of statistical analyses. George Terhanian of Electric Insights has developed a way to use sophisticated statistical techniques to answer some of the most fundamental questions a marketer will ask.

In his article “Hit? Stand? Double? Master’ likely effects’ to make the right call”, George describes his revolutionary process. It is sophisticated behind the scenes, but I like the simplicity in the questions it can address.

He has created a simulation technique that makes sense of complicated data sets. You may measure hundreds of things on a survey and have an excellent profile of the attitudes and behaviors of your customer base. But, where should you focus your investments? This technique demonstrates the likely effects of changes.

As marketers, we cannot directly increase sales. But we can establish and influence attitudes and behaviors that result in sales. Our problem is often to identify which of these attitudes and behaviors to address.

For instance, if I can convince my customer base that my product is environmentally responsible, how many of them can I count on to buy more of my product? The type of simulator described in this article can answer this question, and as a marketer, I can then weigh if the investment necessary is worth the probable payoff.

George created a simulator on some data from a recent Crux Poll. Our poll showed that 17% of Americans trust pollsters. George’s analysis shows that trust in pollsters is directly related to their performance in predicting elections.

Modeling the Crux Poll data showed that if all Americans “strongly agreed” that presidential election polls do a good job of predicting who will win, trust in pollsters/polling organizations would increase by 44 million adults. If Americans feel “extremely confident” that pollsters will accurately predict the 2024 election, trust in pollsters will increase by an additional 40 million adults.

If we are worried that pollsters are untrusted, this suggests that improving the quality of our predictions should address the issue.

Putting research findings in these sorts of terms is what gets our clients’ attention. 

Marketers need this type of quantification because it can plug right into financial plans. Researchers often hear that the reports we provide are not “actionable” enough. There is not much more actionable than showing how many customers would be expected to change their behavior if we successfully invest in a marketing campaign to change an attitude.

Successful marketing is all about putting the probabilities in your favor. Nothing is certain, but as a marketer, your job is to decide where best place your resources (money and time). This type of modeling is a step in the right direction for market researchers.

Associations and Trade Groups for Market Researchers and Pollsters

The market research and polling fields have some excellent trade associations. These organizations help lobby for the industry, conduct studies on issues relating to research, host in-person events and networking opportunities, and post jobs in the market research field. They also host many excellent online seminars. These organizations establish standards for research projects and codes of conduct for their memberships.

Below is a listing of some of the most influential trade groups for market researchers and pollsters. I would recommend that, at minimum, all researchers should get on the email lists of these organizations, as that allows you to see what events and seminars they have coming up. Many of their online seminars are free.

  • ESOMAR. ESOMAR is perhaps the most “worldwide” of all the research trade associations and probably the biggest. ESOMAR was established in 1948 and is headquartered in Europe (Amsterdam). With 40,000 members across 130 countries, it is an influential organization.
  • Insights Association. The Insights Association is U.S.-based. It was created in a merger of two longstanding associations: CASRO and MRA. This organization runs many events and has a certification program for market researchers.
  • Advertising Research Foundation (ARF). ARF concentrates on advertising and media research. ARF puts on a well-known trade show/conference each year and has an important awards program for advertising research, known as the Ogilvy’s. The ARF is likely the most essential trade organization to be a part of if you work in an ad agency or the media or focus on advertising research.
  • Market Research Society. MRS is the U.K. analog to the Insights Association. This organization reaches beyond the U.K. and has some great online courses.
  • The American Association for Public Opinion Research (AAPOR). AAPOR is an influential trade group regarding public opinion polling and pre-election polling. They win the award for longevity, as they have been around since 1947. I consider AAPOR to be the most “academic” of the trade groups, as in addition to researchers and clients, they have quite a few college professors as members. They publish Public Opinion Quarterly, a key academic journal for polling and survey research. AAPOR is a small organization with a large impact.
  • The Research Society. The Research Society is Australia’s key trade association for market researchers.

Many countries have their own trade associations, and there are some associations specific to particular industries, such as pharmaceuticals and health care.

Below are other types of organizations that are not trade associations but are of interest to survey researchers.

  • The Roper Center for Public Opinion Research. The Roper Center is an archive of past polling data, mainly from the U.S. It is currently housed at Cornell University. It can be fascinating to use it to see what American opinion looked like decades ago.
  • The Archive of Market and Social Research (AMSR). AMSR is likely of most interest to U.K. researchers. It is an archive of U.K. history through the lens of polls and market research studies that have been collected.
  • The University of Georgia. The University of Georgia has a leading academic program that trains future market researchers. This university is quite involved in the market research industry and sponsors many exciting seminars. There are some other universities with market research programs, but the University of Georgia is by far the one that is the most tightly connected with the industry.
  • The Burke Institute. The Burke Institute offers many seminars and courses of interest to market research. Many organizations encourage their staff members to take Burke Institute courses.
  • Women in Research (WiRe). WiRe is a group that advances the voice of women in market research. This organization has gained significantly in prominence over the past few years and is doing great work.
  • Green Book. Green Book is a directory of market research firms. Back “in the day,” the Green Book was the printed green directory used by most researchers to find focus group facilities. This organization hosts message boards, conducts industry studies and seminars.
  • Quirk’s. Quirk’s contains interesting articles and runs webinars and conferences.

“Margin of error” sort of explained (+/-5%)

It is now September of an election year. Get ready for a two-month deluge of polls and commentary on them. One thing you can count on is reporters and pundits misinterpreting the meaning behind “margin of error.” This post is meant to simplify the concept.

Margin of error refers to sampling error and is present on every poll or market research survey. It can be mathematically calculated. All polls seek to figure out what everybody thinks by asking a small sample of people. There is always some degree of error in this.

The formula for margin of error is fairly simple and depends mostly on two things: how many people are surveyed and their variability of response. The more people you interview, the lower (better) the margin of error. The more the people you interview give the same response (lower variability), the better the margin of error. If a poll interviews a lot of people and they all seem to be saying the same thing, the margin of error of the poll is low. If the poll interviews a small number of people and they disagree a lot, the margin of error is high.

Most reporters understand that a poll with a lot of respondents is better than one with fewer respondents. But most don’t understand the variability component.

There is another assumption used in the calculation for sampling error as well: the confidence level desired. Almost every pollster will use a 95% confidence level, so for this explanation we don’t have to worry too much about that.

What does it mean to be within the margin of error on a poll? It simply means that the two percentages being compared can be deemed different from one another with 95% confidence. Put another way, if the poll was repeated a zillion times, we’d expect that at least 19 out of 20 times the two numbers would be different.

If Biden is leading Trump in a poll by 8 points and the margin of error is 5 points, we can be confident he is really ahead because this lead is outside the margin of error. Not perfectly confident, but more than 95% confident.

Here is where reporters and pundits mess it up.  Say they are reporting on a poll with a 5-point margin of error and Biden is leading Trump by 4 points. Because this lead is within the margin of error, they will often call it a “statistical dead heat” or say something that implies that the race is tied.

Neither is true. The only way for a poll to have a statistical dead heat is for the exact same number of people to choose each candidate. In this example the race isn’t tied at all, we just have a less than 95% confidence that Biden is leading. In this example, we might be 90% sure that Biden is leading Trump. So, why would anyone call that a statistical dead heat? It would be way better to be reporting the level of confidence that we have that Biden is winning, or the p-value of the result. I have never seen a reporter do that, but some of the election prediction websites do.

Pollsters themselves will misinterpret the concept. They will deem their poll “accurate” as long as the election result is within the margin of error. In close elections this isn’t helpful, as what really matters is making a correct prediction of what will happen.

Most of the 2016 final polls were accurate if you define being accurate as coming within the margin of error. But, since almost all of them predicted the wrong winner, I don’t think we will see future textbooks holding 2016 out there as a zenith of polling accuracy.

Another mistake reporters (and researchers make) is not recognizing that the margin of error only refers to sampling error which is just one of many errors that can occur on a poll. The poor performance of the 2016 presidential polls really had nothing to do with sampling error at all.

I’ve always questioned why there is so much emphasis on sampling error for a couple of reasons. First, the calculation of sampling error assumes you are working with a random sample which in today’s polling world is almost never the case. Second, there are many other types of errors in survey research that are likely more relevant to a poll’s accuracy than sampling error. The focus on sampling error is driven largely because it is the easiest error to mathematically calculate. Margin of error is useful to consider, but needs to be put in context of all the other types of errors that can happen in a poll.

I have more LinkedIn contacts named “Steve” than contacts who are Black

There have been increasing calls for inclusiveness and fairness across America and the world. The issues presented by the MeToo and Black Lives Matter movements affect all sectors of society and the business world. Market research is no exception. Recent events have spurred me to reflect on my experiences and to think about whether the market research field is diverse enough and ready to make meaningful changes. Does market research have structural, systemic barriers preventing women and minorities from succeeding?

My recollections are anecdotal – just one person’s experiences when working in market research for more than 30 years. What follows isn’t based on an industry study or necessarily representative of all researchers’ experiences.

Women in Market Research

When it comes to gender equity in the market research field, my gut reaction is to think that research is a good field for women and one that I would recommend. I reviewed Crux Research’s client base and client contacts. In 15 years, we have worked with about 150 individual research clients across 70 organizations. 110 (73%) of those 150 clients are female. This dovetails with my recollection of my time at a major research supplier. Most of my direct clients there were women.

Crux’s client base is largely mid-career professionals – I’d say our typical client is a research manager or director in his/her 30’s or 40’s. I’d conclude that in my experience, women are well represented at this level.

But, when I look through our list of 70 clients and catalog who the “top” research manager is at these organizations, I find that 42 (60%) of the 70 research VPs and directors are male. And, when I catalog who these research VP’s report into, typically a CMO, I find that 60 (86%) of the 70 individuals are male. To recap, among our client base, 73% of the research managers are female, 40% of the research VPs are female, and 14% of the CMO’s are female.

This meshes with my experience working at a large supplier. While I was there, women were well-represented in our research director and VP roles but there were almost no women represented in the C-suite or among those that report to them. There seem to be a clear but firm glass ceiling in place in market research suppliers and in clients.

Minorities in Market Research

My experience paints a bleaker picture when I think of ethnic minority representation in market research. Of our 150 individual research clients, just 25 (17%) have been non-white and just 3 (2%) have been black. Moving up the corporate ladder, in only 5 (13%) of our 70 clients is the top researcher in the organization non-white and in only 4 (6%) of the 70 companies is the CMO non-white, and none of the CMOs are black. Undoubtedly, we have a long way to go.

A lack of staff diversity in research suppliers and market research corporate staffs is a problem worth resolving for a very important reason: market researchers and pollsters are the folks providing the information to the rest of the world on diversity issues. Our field can’t possibly provide an appropriate perspective to decision makers if we aren’t more diverse. Our lack of diversity affects the conversation because we provide the data the conversation is based upon.  

Non-profits seem to be a notable exception when it comes to ethnic diversity. I have had large non-profit clients that have wonderfully diverse employee bases, to the point where it is not uncommon to attend meetings and Zoom calls where I am the only white male in the session. These non-profits make an effort to recruit and train diverse staffs and their work benefits greatly from the diversity of perspectives this brings. There is a palpable openness of ideas in these organizations. Research clients and suppliers would do well to learn from their example.  

I can’t think of explicit structural barriers that limit the progression of minorities thought the market research ranks, but that just illustrates the problem: the barriers aren’t explicit, they are more subtle and implicit. Which is what makes them so intractable.

We have to make a commitment to develop more diverse employee bases. I worked directly for the CEO of a major supplier for a number of years. One thing I respected about him was he was confident enough in himself that he was not afraid to hire people who were smarter than him or didn’t think like him or came from an entirely different background. It made him unique. In my experience, most hiring managers unintentionally hire “mini-me’s” – younger variants of themselves whom they naturally like in a job interview. Well, if the hiring managers are mostly white males and they are predisposed to hire a lot of “mini-me’s” over time this perpetuates a privilege and is an example of an unintentional, but nonetheless structural bias that limits the progress of women and minorities.

If you don’t think managers tend to hire in their own image, consider a recent Economist article that states “In 2018 there were more men called Steve than there were women among the chief executives of FTSE 100 companies.” I wouldn’t be surprised if there are more market researchers in the US named Steve than there are black market researchers.

To further illustrate that we naturally seek people like ourselves, I reviewed my own LinkedIn contact list. This list is made up of former colleagues, clients, people I have met along the way, etc. It is a good representation of the professional circle I exist within. It turns out that my LinkedIn contact list is 60% female and has 25% non-whites. But, just 3% of my LinkedIn contacts are black. And, yes, I have more LinkedIn contacts named Steve than I have contacts who are black.

This is a problem because as researchers we need to do our best to cast aside our biases and provide an objective analysis of the data we collect. We cannot do that well if we do not have a diverse array of people working on our projects.

Many managers will tell you that they would like to hire a minority for a position but they just don’t get quality candidates applying. This is not taking ownership of the issue. What are you doing to generate candidates in the first place?

It is all too easy to point the finger backwards at colleges and universities and say that we aren’t getting enough qualified candidates of color. And that might be true. MBA programs continue to enroll many more men than women and many more whites than non-whites. They should be taken to task for this. As employers we also need to be making more demands on them to recruit women and minorities to their programs in the first place.

I like that many research firms have come out with supportive statements and financial contributions to relevant causes recently. This is just a first step and needs to be the catalyst to more long-lasting cultural changes in organizations.

We need to share best practices, and our industry associations need to step up and lead this process. Let’s establish relationships with HCBU’s and other institutions to train the next generation of black researchers.

The need to be diverse is also important in the studies we conduct. We need to call more attention to similarities and differences in our analyses – and sample enough minorities in the first place so that we can do this. Most researchers do this already when we have a reason to believe before we launch the study that there might be important differences by race/ethnicity. However, we need to do this more as a matter of course, and become more attuned to highlighting the nuances in our data sets that are driven by race.

Our sample suppliers need to do a better job of recruiting minorities to our studies, and to ensure that the minorities we sample are representative of a wider population. As their clients, we as suppliers need to make more demands about the quality of the minority samples we seek.

We need an advocacy group for minorities in market research. There is an excellent group, Women in Research advocating for women. We need an analogous organization for minorities.

Since I am in research, I naturally think that measurement is key to the solution. I’ve long thought that organizations only change what they can measure. Does your organization’s management team have a formal reporting process that informs them of the diversity of their staff, of their new hires, of the candidates they bring in for interviews? If they do not, your organization is not poised to fix the problem. If your head of HR cannot readily tell you what proportion of your staff is made up of minorities, your firm is likely not paying enough attention.

Researchers will need to realize that their organizations will become better and more profitable when they recruit and develop a more diverse client base. Even though it is the right thing to do, we need to view resolving these issues not solely as altruism. It is in our own self-interest to work on this problem. It is truly the case that if we aren’t part of the solution, we are likely part of the problem. And again, because we are the ones that inform everyone else about public opinion on these issues, we need to lead the way.

My belief is it that this issue will be resolved by Millennials once they get to an age when they are more senior in organizations. Millennials are a generation that is intolerant to unfairness of this sort and notices the subtle biases that add up. They are the most diverse generation in US history. The oldest Millennials are currently in their mid-30’s. In 10-20 years’ time they will be in powerful positions in business, non-profits, education, and government.

Optimistically, I believe Millennials will make a big difference. Pessimistically, I wonder if real change will happen before they are the ones managing suppliers and clients, as thus far the older generations have not shown that they are up to the task.

The myth of the random sample

Sampling is at the heart of market research. We ask a few people questions and then assume everyone else would have answered the same way.

Sampling works in all types of contexts. Your doctor doesn’t need to test all of your blood to determine your cholesterol level – a few ounces will do. Chefs taste a spoonful of their creations and then assume the rest of the pot will taste the same. And, we can predict an election by interviewing a fairly small number of people.

The mathematical procedures that are applied to samples that enable us to project to a broader population all assume that we have a random sample. Or, as I tell research analysts: everything they taught you in statistics assumes you have a random sample. T-tests, hypotheses tests, regressions, etc. all have a random sample as a requirement.

Here is the problem: We almost never have a random sample in market research studies. I say “almost” because I suppose it is possible to do, but over 30 years and 3,500 projects I don’t think I have been involved in even one project that can honestly claim a random sample. A random sample is sort of a Holy Grail of market research.

A random sample might be possible if you have a captive audience. You can random sample some the passengers on a flight or a few students in a classroom or prisoners in a detention facility. As long as you are not trying to project beyond that flight or that classroom or that jail, the math behind random sampling will apply.

Here is the bigger problem: Most researchers don’t recognize this, disclose this, or think through how to deal with it. Even worse, many purport that their samples are indeed random, when they are not.

For a bit of research history, once the market research industry really got going the telephone random digit dial (RDD) sample became standard. Telephone researchers could randomly call land line phones. When land line telephone penetration and response rates were both high, this provided excellent data. However, RDD still wasn’t providing a true random, or probability sample. Some households had more than one phone line (and few researchers corrected for this), many people lived in group situations (colleges, medical facilities) where they couldn’t be reached, some did not have a land line, and even at its peak, telephone response rates were only about 70%. Not bad. But, also, not random.

Once the Internet came of age, researchers were presented with new sampling opportunities and challenges. Telephone response rates plummeted (to 5-10%) making telephone research prohibitively expensive and of poor quality. Online, there was no national directory of email addresses or cell phone numbers and there were legal prohibitions against spamming, so researchers had to find new ways to contact people for surveys.

Initially, and this is still a dominant method today, research firms created opt-in panels of respondents. Potential research participants were asked to join a panel, filled out an extensive demographic survey, and were paid small incentives to take part in projects. These panels suffer from three response issues: 1) not everyone is online or online at the same frequency, 2) not everyone who is online wants to be in a panel, and 3) not everyone in the panel will take part in a study. The result is a convenience sample. Good researchers figured out sophisticated ways to handle the sampling challenges that result from panel-based samples, and they work well for most studies. But, in no way are they a random sample.

River sampling is a term often used to describe respondents who are “intercepted” on the Internet and asked to fill out a survey. Potential respondents are invited via online ads and offers placed on a range of websites. If interested, they are typically pre-screened and sent along to the online questionnaire.

Because so much is known about what people are doing online these days, sampling firms have some excellent science behind how they obtain respondents efficiently with river sampling. It can work well, but response rates are low and the nature of the online world is changing fast, so it is hard to get a consistent river sample over time. Nobody being honest would ever use the term “random sampling” when describing river samples.

Panel-based samples and river samples represent how the lion’s share of primary market research is being conducted today. They are fast and inexpensive and when conducted intelligently can approximate the findings of a random sample. They are far from perfect, but I like that the companies providing them don’t promote them as being random samples. They involve some biases and we deal with these biases as best we can methodologically. But, too often we forget that they violate a key assumption that the statistical tests we run require: that the sample is random. For most studies, they are truly “close enough,” but the problem is we usually fail to state the obvious – that we are using statistical tests that are technically not appropriate for the data sets we have gathered.

Which brings us to a newer, shiny object in the research sampling world: ABS samples. ABS (addressed-based samples) are purer from a methodological standpoint. While ABS samples have been around for quite some time, they are just now being used extensively in market research.

ABS samples are based on US Postal Service lists. Because USPS has a list of all US households, this list is an excellent sampling frame. (The Census Bureau also has an excellent list, but it is not available for researchers to use.) The USPS list is the starting point for ABS samples.

Research firms will take the USPS list and recruit respondents from it, either to be in a panel or to take part in an individual study. This recruitment can be done by mail, phone, or even online. They often append publicly-known information onto the list.

As you might expect, an ABS approach suffers from some of the same issues as other approaches. Cooperation rates are low and incentives (sometimes large) are necessary. Most surveys are conducted online, and not everyone in the USPS list is online or has the same level of online access. There are some groups (undocumented immigrants, homeless) that may not be in the USPS list at all. Some (RVers, college students, frequent travelers) are hard to reach. There is evidence that ABS approaches do not cover rural areas as well as urban areas. Some households use post office boxes and not residential addresses for their mail. Some use more than one address. So, although ABS lists cover about 97% of US households, the 3% that they do not cover are not randomly distributed.

The good news is, if done correctly, the biases that result from an ABS sample are more “correctable” than those from other types of samples because they are measurable.

A recent Pew study indicates that survey bias and the number of bogus respondents is a bit smaller for ABS samples than opt-in panel samples.

But ABS samples are not random samples either. I have seen articles that suggest that of all those approached to take part in a study based on an ABS sample, less than 10% end up in the survey data set.

The problem is not necessarily with ABS samples, as most researchers would concur that they are the best option we have and come the closest to a random sample. The problem is that many firms that are providing ABS samples are selling them as “random samples” and that is disingenuous at best. Just because the sampling frame used to recruit a survey panel can claim to be “random” does not imply that the respondents you end up in a research database constitute a random sample.

Does this matter? In many ways, it likely does not. There are biases and errors in all market research surveys. These biases and errors vary not just by how the study was sampled, but also by the topic of the question, its tone, the length of the survey, etc. Many times, survey errors are not the same throughout an individual survey. Biases in surveys tend to be “unknown knowns” – we know they are there, but aren’t sure what they are.

There are many potential sources of errors in survey research. I am always reminded of a quote from Humphrey Taylor, the past Chairman of the Harris Poll who said “On almost every occasion when we release a new survey, someone in the media will ask, “What is the margin of error for this survey?” There is only one honest and accurate answer to this question — which I sometimes use to the great confusion of my audience — and that is, “The possible margin of error is infinite.”  A few years ago, I wrote a post on biases and errors in research, and I was able to quickly name 15 of them before I even had to do an Internet search to learn more about them.

The reality is, the improvement in bias that is achieved by an ABS sample over a panel-based sample is small and likely inconsequential when considered next to the other sources of error that can creep into a research project. Because of this, and the fact that ABS sampling is really expensive, we tend to only recommend ABS panels in two cases: 1) if the study will result in academic publication, as academics are more accepting of data that comes from and ABS approach, and 2) if we are working in a small geography, where panel-based samples are not feasible.

Again, ABS samples are likely the best samples we have at this moment. But firms that provide them are often inappropriately portraying them as yielding random samples. For most projects, the small improvements in bias they provide is not worth the considerable increased budget and increased study time frame, which is why, for the moment, ABS samples are currently used in a small proportion of research studies. I consider ABS to be “state of the art” with the emphasis on “art” as sampling is often less of a science than people think.

Should you use DIY market research tools?

A market research innovation has occurred over the past decade that is talked about in hushed tones among research suppliers:  the rise of DIY market research tools. Researchers and clients need to become more educated on what these DIY tools are and when it is appropriate to used them.

DIY tools come in a number of flavors. At their core, they allow anybody to log into a system, author a survey, select sample parameters, and hit “go.” Many also provide the ability to tabulate data and graph results. These tools reduce the complexity of fielding studies. For the most part, these tools are created by outside panel and research technology companies but some end clients have invested in their own tools.

Many research suppliers view DIY tools as an existential threat. After all, if clients can do all this themselves what do they need us for? Will our fielding and programming departments become obsolete? Will we have a large portion of what we do automated?

Maybe. But more likely our fielding and programming departments will become smaller and have to adapt to a changing technological world.

There is a clear analogy here to DIY household projects. The tools and materials needed for most home improvement projects are available at big box retailers. Some homeowners are well-equipped to take on projects themselves, others are not, and the key to a successful project is often understanding when it is important to call for professional help. The same is true for market research projects.

Where the analogy fails is when you take on a project you aren’t equipped to handle. If it is a home project you will probably discover that you got in a bit over your head along the way. In market research, however, you can complete an entire project that has serious errors in it but never really notice. The project will result in sub-optimal decision making and nobody may really notice.

In days gone by, the decision of whether to use a research supplier or not was straightforward. If the project was meaningful or complex, clients used suppliers. For many projects, the choice used to be between using a supplier or not doing the project at all. The rise of DIY tools has changed that.

Here are some instances where DIY research makes sense:

  • If the project is relatively simple. By simple, we mean from both a questionnaire design and a sampling perspective.
  • If the risk of making a suboptimal decision based on the information is low. Perhaps the best aspect of DIY tools is they permit clients to research issues that otherwise may have gone unresearched because of time and budget considerations.
  • When getting it done quickly is important. For many projects, there is something to be said for getting it 90% right and getting it done today rather than taking months to get it perfect.
  • If you have someone with supplier-side experience on staff. Suppliers are likely to be a bit more attuned to the nuances of study design and may notice mistakes others might miss.
  • If you have thought through the potential limitations of the DIY approach and have communicated this to your internal client.
  • When you are using the DIY project to pre-test or pilot a study. This is an excellent use of DIY tools: to be sure your questioning and scales are going to work before committing significant resources to a project. A DIY project can make the subsequent project more efficient.

Here are cases when we would caution against using DIY tools:

  • If a consequential decision will be made based on the results. Having the backing of a third-party supplier is important in this case and the investment is likely worth it.
  • When research results need to motivate people internally. Internal decision makers will typically listen more to research results if the study was conducted by a third-party.
  • When a broader perspective is needed. As a client, you know your firm and industry better than most suppliers will. But there are many times when having a broader perspective on a project provides substantial value to it.
  • If the sampling is complicated. If your target audience is obscure and hard to define in a few words, suppliers can be very helpful in getting your sampling right. In a previous post we mention that it is the sampling aspects of projects that most clients don’t think through enough. We have found that the most serious mistakes made in market research deal with sampling, and often these mistakes are hard to notice.
  • If you are conducting a business-to-business study. DIY sampling resources aren’t yet of the same quality for b-to-b research as they are for consumer studies.

DIY studies clearly have their place. They will augment current studies in some cases and replace them in others. I don’t see them as a threat to the highly-customized types of studies Crux Research tends to conduct. Market research spending will continue to grow slowly, but less will be spent on data collection and more on higher value-added aspects of projects.

In the 30 years I have worked in research, the cost of data collection has dropped considerably – I’d say it is about one-third what it used to be. But, during this time the price of research projects has increased. The implication is that clients have come to value the consultative aspects of studies more and have become more reliant on their suppliers to do things that previously clients did for themselves.

That presents a bit of a conundrum: clients are outsourcing more to suppliers at a time when tools are being developed that allow them to do many projects without a supplier. For many clients, money and time would be saved by hiring someone on staff that knows how to use these tools recognizes when a third-party supplier is necessary.

Did Apple just kill telephone market research?

A recent issue of The Economist contained an article that describes a potential threat to the accuracy of opinion polling. The latest iPhones have a software feature that doesn’t just block robocalls but sends all calls from unknown callers automatically to voice mail. This feature combats unwanted calls on mobile phones.

Matching sampling frames to populations of interest is increasingly difficult to accomplish in survey research, particularly telephone studies. I will always remember my first day on the job in 1989 when my supervisor was teaching me how to bid projects. The spreadsheet we used assumed our telephone polls would have a 60% cooperation rate. So, at that time about 6 in 10 phone calls we made resulted in a willing respondent. Currently, telephone studies rarely achieve a cooperation rate above 5%. That is 1 in 20 calls. If you are lucky.

The Do Not Call Registry took effect in 2003. At this time, most survey research was still being conducted by telephone (online research was growing but still represented only about 20% of the market research industry’s revenues). Researchers were initially relieved that market research and polls were exempt from the law but in the end that didn’t matter. People stopped cooperating with telephone studies because they thought they had opted out of research calls when they signed up for the Registry. Response rates plummeted.

The rise of mobile phones caused even more headaches for telephone researchers. There was initially no great way to generate random numbers of cell phones in the same way that could be done for land lines and publicly-available directories of cell phone numbers did not exist. For quite some time, telephone studies were underrepresenting mobile phone users and had no great solution for how to interview respondents who did not even have a land line. Eventually, the industry figured this out and methods for including mobile phones became standard.

This new development of automatically routing mobile calls to voice mail could well signify the end of telephone-based research. If consumers like this feature on iPhones it won’t be long until Android-based phones do the same thing. It will preclude pollsters from effectively reaching mobile-only households. Believe it or not, about 45% of US households still have a land line, but the 55% who do not skew young, urban, and liberal.

Pollsters will figure this out and will oversample mobile only households and weight them up in samples. But that won’t really fix the problem. Samples will miss those that have the latest phones and will eventually miss everybody once all current phones are replaced. Oversampling and weighting can help balance under-represented groups, but can’t fix a problem when a group is not represented at all. Weighting can actually magnify biases in samples.

Implications to this?  Here are a few:

  1. More polls and market research project will be conducted online. This is a good thing as there is evidence that in the 2016 election the online polls were more accurate than the telephone polls. It is hard to believe, but we are at a stage where telephone polls are almost always slower, more expensive, and less accurate than their online counterparts.
  2. Researchers will use more mixed samples, using both telephone and online. In our view this tends to be needlessly complicated and introduces mode effects into these samples. We tend to only recommend mixed-mode data collection in business-to-business projects, where we use the phone to screen to a qualified respondent and then send the questionnaire electronically.
  3. Costs of telephone polls will go up. They are already almost criminally expensive and this will get even worse. For those not in the know, the cost per interview for a telephone poll is often 20 to 30 times the cost of an online interview.
  4. Address Based Samples (ABS) will gain in popularity. As telephone response rates decline, systematic biases in telephone samples increase. ABS, when properly operationalized, is a good alternative (although ABS has its limitations as well). ABS still isn’t really probability sampling, but it is the closest thing we have.
  5. The increased cost of telephone polls will spur even more investment in online panels. The quality of online research will be better off because of it. If there is a silver lining for researchers, this is probably it.

Technology has always tended to move faster than the market research industry has been able to adapt to it, probably because researchers have an academic mindset (thorough, but slow). Research methodologists always seem to eventually come up with a solution, but not always quickly. For now, we’d recommend against trusting any opinion poll that is based on a telephone sample, unless the researchers behind it have specifically made a case for how they are going to address this new issue of software blocking their calls to mobile phones. The good news is push polls and robo polls will soon become almost impossible to conduct.

Among college students, Bernie Sanders is the overwhelming choice for the Democratic nomination

Crux Research poll of college students shows Sanders at 23%, Biden at 16%, and all other candidates under 10%

ROCHESTER, NY – October 10, 2019 – Polling results released today by Crux Research show that if it was up to college students, Bernie Sanders would win the Democratic nomination the US Presidency. Sanders is the favored candidate for the nomination among 23% of college students compared to 16% for Joe Biden. Elizabeth Warren is favored by 8% of college students followed by 7% support for Andrew Yang.

  • Bernie Sanders: 23%
  • Joe Biden: 16%
  • Elizabeth Warren: 8%
  • Andrew Yang: 7%
  • Kamala Harris: 6%
  • Beto O’Rourke: 5%
  • Pete Buttigieg: 4%
  • Tom Steyer: 3%
  • Cory Booker: 3%
  • Michael Bennet: 2%
  • Tulsi Gabbard: 2%
  • Amy Klobuchar: 2%
  • Julian Castro: 1%
  • None of these: 5%
  • Unsure: 10%
  • I won’t vote: 4%

The poll also presented five head-to-head match-ups. Each match-up suggests that the Democratic candidate currently has a strong edge over President Trump, with Sanders having the largest edge.

  • Sanders versus Trump: 61% Sanders; 17% Trump; 12% Someone Else; 7% Not Sure; 3% would not vote
  • Warren versus Trump: 53% Warren; 18% Trump; 15% Someone Else; 9% Not Sure; 5% would not vote
  • Biden versus Trump: 51% Biden; 18% Trump; 19% Someone Else; 8% Not Sure; 4% would not vote
  • Harris versus Trump: 48% Harris; 18% Trump; 20% Someone Else; 10% Not Sure; 4% would not vote
  • Buttigieg versus Trump: 44% Buttigieg; 18% Trump; 22% Someone Else; 11% Not Sure; 5% would not vote

The 2020 election could very well be determined on the voter turnout among young people, which has traditionally been much lower than among older age groups.


This poll was conducted online between October 1 and October 8, 2019. The sample size was 555 US college students (aged 18 to 29). Quota sampling and weighting were employed to ensure that respondent proportions for age group, sex, race/ethnicity, and region matched their actual proportions in the US college student population.

This poll did not have a sponsor and was conducted and funded by Crux Research, an independent market research firm that is not in any way associated with political parties, candidates, or the media.

All surveys and polls are subject to many sources of error. The term “margin of error” is misleading for online polls, which are not based on a probability sample which is a requirement for margin of error calculations. If this study did use probability sampling, the margin of error would be +/-4%.

About Crux Research Inc.
Crux Research partners with clients to develop winning products and services, build powerful brands, create engaging marketing strategies, enhance customer satisfaction and loyalty, improve products and services, and get the most out of their advertising.

Using quantitative and qualitative methods, Crux connects organizations with their customers in a wide range of industries, including health care, education, consumer goods, financial services, media and advertising, automotive, technology, retail, business-to-business, and non-profits.
Crux connects decision makers with customers, uses data to inspire new thinking, and assures clients they are being served by experienced, senior level researchers who set the standard for customer service from a survey research and polling consultant.

To learn more about Crux Research, visit

Visit the Crux Research Website

Enter your email address to follow this blog and receive notifications of new posts by email.