Archive for the 'Marketing' Category

Less useful research questions

Questionnaire “real estate” is limited and valuable. Most surveys fielded today are too long and this causes problems with respondent fatigue and trust. Researchers tend to start the questionnaire design process with good intent and aim to keep survey experiences short and compelling for respondents. However, it is rare to see a questionnaire get shorter as it undergoes revision and review, and many times the result is impossibly long surveys.

One way to guard against this is to be mindful. All questions included should have a clear purpose and tie back to study objectives. Many times, researchers include some questions and options simply out of habit, and not because these questions will add value to the project.

Below are examples of question types that, more often or not, add little to most questionnaires. These questions are common and used out of habit. There are certainly exceptions when it makes sense to include these questions, but for the most part we advise against using them unless there is a specific reason to include them.

Marital status

Somewhere along the way, asking a respondent’s marital status became standard on most consumer questionnaires. Across thousands of studies, I can only recall a few times when I have actually used it for anything. It is appropriate to ask if it is relevant. Perhaps your client is a jewelry company or in the bridal industry. Or, maybe you are studying relationships. However, I would nominate marital status as being the least used question in survey research history.

Other (specify)

Many multiple response questions ask a respondent to select all that apply from a list, and then as a final option will have “other.” Clients constantly pressure researchers to leave a space for respondents to type out what this “other” option is. We rarely look at what they type in. I tell clients that if we expect a lot of respondents to select the other option, it probably means that we have not done a good job at developing the list. It may also mean that we should be asking the question in an open-ended fashion. Even when it is included, most of the respondents who select other will not type anything into the little box anyway.

Don’t Know Options

We recently composed an entire post about when to include a Don’t Know option on a question. To sum it up, the incoming assumption should be that you will not use a Don’t Know option unless you have an explicit reason to do so. Including Don’t Know as an option can make a data set hard to analyze. However, there are exceptions to this rule, as Don’t Know can be an appropriate choice. That said, it is overused on surveys currently.

Open-Ends

The transition from telephone to online research has completely changed how researchers can ask open-ended questions. In the telephone days, we could pose questions that were very open-ended because we had trained interviewers who could probe for meaningful answers. With online surveys, open-ended questions that are too loose rarely produce useful information. Open-ends need to be specific and targeted. We favor the inclusion of just a handful of open-ends in each survey, and that they are a bit less “open-ended” than what has been traditionally asked.

Grid questions with long lists

We have all seen these. These are long lists of items that require a scaled response, perhaps a 5-point agree/disagree scale. The most common abandon point on a survey is the first time a respondent encounters a grid question with a long list. Ideally, these lists are about 4 to 6 items and there are no more than two or three of them on a questionnaire.

We currently field a study that has a list like this with 28 items in it. There is no way we are getting good information from this question and we are fatiguing the respondent for the remainder of the survey.

Specifying time frames

Survey research often seeks to find out about a behavior across a specified time frame. For instance, we might want to know if a consumer has used a product in the past day, past week, past month, etc. The issue here is not so much the time frame, it is when we consider the responses to be literal. I have seen clients take past day usage and multiply it by 365 and assume that will equate to past year usage. Technically and mathematically, that might be true, but it isn’t how respondents react to questions.

In reality, it is likely accurate to ask if a respondent has done something in the past day. But, once the time frames get longer, we are really asking about “ever” usage. It depends a bit on the purchase cycle of the product and its cost, but for most products, asking if they have used in the past month, 6 months, year, etc. will yield similar responses.

Some researchers work around this by just asking “ever used” and “recently used.” There are times when that works, but we tend to set a reasonable time frame for recent use and go with that, typically within the past week.

Household income

Researchers have asked household income as long as the survey research field has been around. There are at least three serious problems with it. First, many respondents are not knowledgeable about what their household income is. Most households have a “family CFO” who takes the lead on financial issues, and even this person often will not know what the family income is. 

Second, the categories chosen affect the response to the income question, indicating just how unstable it is. Asking household income in say, ten categories versus five categories will not result in comparable data. Respondents tend to assume the middle of the range given is normal, and respond using that as a reference point.

Third, and most importantly, household income is a lousy measure of socio-economic status (SES). Many young people have low annual incomes but a wealthy lifestyle as they are still being supported by their parents. Many older people are retired and may have almost non-existent incomes, yet live a wealthy lifestyle off of their savings. Household income tends to only be a reasonable measure of SES for respondents aged about 30 to 60,

There are better measures of SES. Education level can work, and a particularly good question is to ask the respondent about their mother’s level of education, which has been shown to correlate strongly with SES. We also ask about their attitudes towards their income – whether they have all the money they need, just enough, or if they struggle to meet basic expenses.

Attention spans are getting shorter and as more and more surveys are being completed on mobile devices there are plenty of distractions as respondents answer questionnaires. Engage them, get their attention, and keep the questionnaire short. There may be no such thing as a dumb question, but there are certainly questions that when asked on a survey do not yield useful information.

Should you include a “Don’t Know” option on your survey question?

Questionnaire writers construct a bridge between client objectives and a line of questioning that a respondent can understand. This is an underappreciated skill.

The best questionnaire writers empathize with respondents and think deeply about tasks respondents are asked to perform. We want to strike a balance between the level of cognitive effort required and a need to efficiently gather large amounts of data. If the cognitive effort required is too low, the data captured is not of high quality. If it is too high, respondents get fatigued and stop attending to our questions.

One of the most common decisions researchers have to make is whether or not to allow for a Don’t Know (DK) option on a question. This is often a difficult choice, and the correct answer on whether to include a DK option might be the worst possible answer: “It depends.”

Researchers have genuine disagreements about the value of a DK option. I lean strongly towards not using DK’s unless there is a clear and considered reason for doing so.

Clients pay us to get answers from respondents and to find out what they know, not what they don’t know. Pragmatically, whenever you are considering adding a DK option your first inclination should be that you perhaps have not designed the question well. If a large proportion of your respondent base will potentially choose “don’t know,” odds are high that you are not asking a good question to begin with, but there are exceptions.

If you get in a situation where you are not sure if you should include a DK option, the right thing to do is to think broadly and reconsider your goal: why are you asking the question in the first place? Here is an example which shows how the DK decision can actually be more complicated than it first appears.

We recently had a client that wanted us to ask a question similar to this: “Think about the last soft drink you consumed. Did this soft drink have any artificial ingredients?”

Our quandary was whether we should just ask this as a Yes/No question or to also give the respondent a DK option. There was some discussion back and forth, as we initially favored not including DK, but our client wanted it.

Then it dawned on us that whether or not to include DK depended on what the client wanted to get out of the question. On one hand, the client might want to truly understand if the last soft drink consumed had any artificial ingredients in it, which is ostensibly what the question asks. If this was the goal, we felt it was necessary to better educate the respondent on what an “artificial ingredient” was so they could provide an informed answer and so all respondents would be working from a common definition. Or, alternatively, we could ask for the exact brand and type of soft drink they consumed and then on the back-end code which ones have artificial ingredients and which do not, and thus get a good estimate for the client.

The other option was to realize that respondents might have their own definitions of “artificial ingredients” that may or may not match our client’s definition. Or, they may have no clue what is artificial and what is not.

In the end, we decided to use the DK option in this case because understanding how many people are ignorant to artificial ingredients fit well with our objectives. When we pressed the client, we learned that they wanted to document this ambiguity. If a third of consumers don’t know whether or not their soft drinks have artificial ingredients in them, this would be useful information for our client to know.

This is a good example on how a seemingly simple question can have a lot of thinking behind it and how it is important to contextualize this reasoning when reporting results. In this case, we are not really measuring whether people are drinking soft drinks with artificial ingredients. We are measuring what they think they are doing, which is not the same thing and likely more relevant from a marketing point-of-view.

There are other times when a DK option makes sense to include. For instance, some researchers will conflate the lack of an option (a DK response) with a neutral opinion and these are not the same thing. For example, we could be asking “how would you rate the job Joe Biden is doing as President?” Someone who answers in the middle of the response scale likely has a considered, neutral opinion of Joe Biden. Someone answering DK has not considered the issue and should not be assumed to have a neutral opinion of the president. This is another case where it might make sense to use DK.

However, there are probably more times when including a DK option is a result of lazy questionnaire design than any deep thought regarding objectives. In practice, I have found that it tends to be clients who are inexperienced in market research that press hardest to include DK options.

There are at least a couple of serious problems with including DK options on questionnaires. The first is “satisficing” – which is a tendency respondents have to not place a lot of effort on responding and instead choose the option that requires the least cognitive effort. The DK option encourages satisficing. A DK option also allows respondents to disengage with the survey and can lead to inattention on subsequent items.

DK responses create difficulties when analyzing data. We like to look at questions on a common base of respondents, and that becomes hard to comprehend when respondents choose DK on some questions but not others. Including DK makes it harder to compare results across questions. DK options also limit the ability to use multivariate statistics, as a DK response does not fit neatly on a scale.

Critics would say that researchers should not force respondents to express and opinion they do not have and therefore should provide DK options. I would counter by saying that if you expect a substantial amount of people to not have an opinion, odds are high you should reframe the question and ask them about something they do know about. It is usually (but not always) the case that we want to find out more about what people know than what they don’t know.

“Don’t know” can be a plausible response. But, more often than not, even when it is a plausible response if we feel a lot of people will choose it, we should reconsider why we are asking the question. Yes, we don’t want to force people to express an option they don’t have. But rather than include DK, it is better to rewrite a question to be more inclusive of everybody.

As an extreme example, here is a scenario that shows how a DK can be designed out of a question:

We might start with a question the client provides us: “How many minutes does your child spend doing homework on a typical night?” For this question, it wouldn’t take much pretesting to realize that many parents don’t really know the answer to this, so our initial reaction might be to include a DK option. If we don’t, parents may give an uninformed answer.

However, upon further thought, we should realize that we may not really care about how many minutes the child spends on homework and we don’t really need to know whether the parent knows this precisely or not. Thinking even deeper, some kids are much more efficient in their homework time than others, so measuring quantity isn’t really what we want at all. What we really want to know is, is the child’s homework level appropriate and effective from the parent’s perspective?

This probing may lead us down a road to consider better questions, such as “in your opinion, does your child have too much, too little, or about the right amount of homework?” or “does the time your child spends on homework help enhance his/her understanding of the material?” This is another case when thinking more about why we are asking the question tends to result in better questions being posed.

This sort of scenario happens a lot when we start out thinking we want to ask about a behavior, when what we really want to do is ask about an attitude.

The academic research on this topic is fairly inconclusive and sometimes contradictory. I think that is because academic researchers don’t consider the most basic question, which is whether or not including DK will better serve the client’s needs. There are times that understanding that respondents don’t know is useful. But, in my experience, more often than not if a lot of respondents choose DK it means that the question wasn’t designed well. 

Which quality control checks questions should you use in your surveys?

While it is no secret that the quality of market research data has declined, how to address poor data quality is rarely discussed among clients and suppliers. When I started in market research more than 30 years ago, telephone response rates were about 60%. Six in 10 people contacted for a market research study would choose to cooperate and take our polls. Currently, telephone response rates are under 5%. If we are lucky, 1 in 20 people will take part. Online research is no better, as even from verified customer lists response rates are commonly under 10% and even the best research panels can have response rates under 5%.

Even worse, once someone does respond, a researcher has to guard against “bogus” interviews that come from scripts and bots, as well as individuals who are cheating on the survey to claim the incentives offered. Poor-quality data is clearly on the rise and is an existential threat to the market research industry that is not being taken seriously enough.

Maximizing response requires a broad approach with tactics deployed throughout the process. One important step is to cleanse each project of bad quality respondents. Another hidden secret in market research is that researchers routinely have to remove anywhere from 10% to 50% of respondents from their database due to poor quality.

Unfortunately, there is no industry standard way of doing this – of identifying poor-quality respondents. Every supplier sets their own policies. This is likely because there is considerable variability in how respondents are sourced for studies, and a one-size-fits-all approach might not be possible, and some quality checks depend on the specific topic of the study. Unfortunately, researchers are left to largely fend for themselves when trying to come up with a process for how to remove poor quality respondents from their data.

One of the most important ways to guard against poor quality respondents is to design a compelling questionnaire to begin with. Respondents will attend to a short, relevant survey. Unfortunately, we rarely provide them with this experience.

We have been researching this issue recently in an effort to come up with a workable process for our projects. Below, we share our thoughts. The market research industry needs to work together on this issue, as when one of us removes a bad respondent from a database in helps the next firm with their future studies.

There is a practical concern for most studies – we rarely have room for more than a handful of questions that relate to quality control. In addition to speeder and straight-line checks, studies tend to have room for about 4-5 quality control questions. With the exception of “severe speeders” as described below, respondents will be automatically removed if they fail three or more of the checks. We use a “three strikes and you’re out” rule to remove respondents. If anything, this is probably too conservative, but we’d rather err on the side of retaining some bad quality respondents in than inadvertently removing some good quality ones.

When possible, we favor checks that can be done programmatically, without human intervention, as that keeps fielding and quota management more efficient. To the degree possible, all quality check questions should have a base of “all respondents” and not be asked of subgroups.

Speeder Checks

We aim to set up two criteria: “severe” speeders are those that complete the survey in less than one-third of the median time. These respondents are automatically tossed. “Speeders” are those that take between one-third and one-half of the median time, and these respondents are flagged.

We also consider setting up timers within the survey – for example, we may place timers on a particularly long grid question or a question that requires substantial reading on the part of the respondent. Note that when establishing speeder checks it is important to use the median length as a benchmark and not the mean. In online surveys, some respondents will start a survey and then get distracted for a few hours and come back to it, and this really skews the average survey length. Using the median gets around that.

Straight Line Checks

Hopefully, we have designed our study well and do not have long grid type questions. However, more often than not these types of questions find their way into questionnaires.  For grids with more than about six items, we place a straight-lining check – if a respondent chooses the same response for all items in the grid, they are flagged.

Inconsistent Answers

We consider adding two question that check for inconsistent answers. First, we re-ask a demographic question from the screener near the end of the survey. We typically use “age” as this question. If the respondent doesn’t choose the same age in both questions, they are flagged.

In addition, we try to find an attitudinal question that is asked that we can re-ask in the exact opposite way. For instance, if earlier we asked “I like to go to the mall” on a 5-point agreement scale, we will also ask the opposite: “I do not like to go to the mall” on the same scale. Those that answer the same for both are flagged. We try to place these two questions a few minutes apart in the questionnaire.

Low Incidence items

This is a low attentiveness flag. It is meant to catch people who say they do really unlikely things and also catch people who say they don’t do likely things because they are not really paying attention to the questions we pose. We design this question specific to each survey and tend to ask what respondents have done over the past weekend. We like to have two high incidence items (such as “watched TV,” or “rode in a car”), 4 to 5 low incidence items (such as “flew in an airplane,” “read an entire book,” “played poker”) and one incredibly low incidence item (such as “visited Argentina”).  Respondents are flagged if they didn’t do at least one of our high incidence items, if they said they did more than two of our low incidence items, or if they say they did our incredibly low incidence item.

Open-ended check

We try to include this one in all studies, but sometimes have to skip it if the study is fielding on a tight timeframe because it involves a manual process. Here, we are seeing if a respondent provides a meaningful response to an open-ended question. Hopefully, we can use a question that is already in the study for this, but when we cannot we tend to use one like this: “Now I’d like to hear your opinions about some other things. Tell me about a social issue or cause that you really care about.  What is this cause and why do you care about it?” We are manually looking to see if they provide an articulate answer and they are flagged if they do not.

Admission of inattentiveness

We don’t use this one as a standard, but are starting to experiment with it. As the last question of the survey, we can ask respondents how attentive they were. This will suffer from a large social desirability bias, but we will sometimes directly ask them how attentive they were when taking the survey, and flag those that say they did not pay attention at all.

Traps and misdirects

I don’t really like the idea of “trick questions” – there is research that indicates that these types of questions tend to trap too many “good” respondents. Some researchers feel that these questions lower respondent trust and thus answer quality. That seems to be enough to recommend against this style of question. The most common types I have seen ask a respondent to select the “third choice” below no matter what, or to “pick the color from the list below,” or “select none of the above.” We counsel against using these.

Comprehension

This was recommended by a research colleague and was also mentioned by an expert in a questionnaire design seminar we attended. We don’t use this as a quality check, but like to use it during a soft-launch period. The question looks like this: “Thanks again for taking this survey.  Were there any questions on this survey you had difficulty with or trouble answering?  If so, it will be helpful to us if you let us know what those problems were in the space below.” This is a useful question, but we don’t use it as a quality check per se.

Preamble

I have mixed feelings on this type of quality check, but we use it when we can phrase it positively. A typical wording is like this: “By clicking yes, you agree to continue to our survey and give your best effort to answer 10-15 minutes of questions. If you speed through the survey or otherwise don’t give a good effort, you will not receive credit for taking the survey.”

This is usually one of the first questions in the survey. The argument I see against this is it sets the respondent up to think we’ll be watching them and that could potentially affect their answers. Then again, it might affect them in a good way if it makes them attend more.

I prefer a question that takes a gentler, more positive approach – telling respondents we are conducting this for an important organization, that their opinions will really matter, promise them confidentiality, and then ask them to agree to give their best effort, as opposed to lightly threatening them as this one does.

Guarding against bad respondents has become an important part of questionnaire design, and it is unfortunate that there is no industry standard on how to go about it. We try to build in some quality checks that will at least spot the most egregious cases of poor quality. This is an evolving issue, and it is likely that what we are doing today will change over time, as the nature of market research changes.

Should all college majors pay the same tuition?

Despite all that is written about the costs of higher education and how student debt is crippling an entire generation, college remains a solid investment for most students. The Bureau of Labor Statistics indicates that people with bachelor’s degrees earn about $1,173 on average each week while those with only high school diplomas earn an average of $712 per week. That is a difference of $461 per week, about $24,000 per year, and about $958,880 over a 40-year working lifetime. On average, four-year college graduates literally are about a million dollars better off in their lifetime than those that stop their education after high school.

This calculation suffers from a selection bias, as individuals that choose to go to college likely have higher earnings potential that those that do not, independent of their education, so it is not appropriate to credit the colleges entirely for the million dollar increase in value. But, at pretty much any tuition level it would be hard to argue that college does not pay off for most graduates.

This helps put the student debt debate in perspective. The average student debt is about $30,000. A typical U.S. college student goes $30,000 in debt to gain a credential that will earn an average of about $1,000,000 more over his/her lifetime. College costs are far too high, have grown considerably faster that colleges’ ability to increase value, and limit many worthy students from being able to furthering their education. Yet, college remains a stellar asset for most.

These calculations concentrate on an “average student” and much can be lost by doing that. About 1 in 5 college graduates carries more than $50,000 in loans. About 1 in 20 has more than $100,000 in loans. Not all college graduates make a million dollars more over their lifetimes. Plenty of students slip through the cracks and many are underemployed because of a mismatch between their training and what employers demand.

Many young people are in financial trouble because college is not an investment that is paying back quickly enough for them. There are too many students who begin college, take on debt, and never graduate and gain the credential that enhances their earning power. The most hidden statistic in America may be that only about 60% of those who enroll in college end up graduating.

There is an enormous disparity in the average starting salary for college graduates depending on their major and their college. When thinking of the financial aspects of college, parents and students would be wise to look more at the debt to earnings ratio rather than concentrate solely on the costs of college. That is, what will an expected first year salary be and what will the expected college debt be?

A rule of thumb is to try to get this ratio as far under 1.0 as possible, and to not let it go over 1.0. This means that students should seek to have loans that do not total more than their expected first year salary, and hopefully loans that are just a fraction of their first-year salary.

Data from the Department of Education’s College Scorecard shows average student debt and average first year salary by college and by major. What is striking is how much variability there is on the salary part and how little there is on the debt part. Broadly speaking, salaries vary widely by college and major, but the debt students end up with does not vary nearly as much.

Suppose you owned a business and two customers walked into your door. For customer A, you provide a service that is worth twice as much as what you provide to customer B. Would you charge both customers the same amount? Probably not. They would not expect you to even if it cost you the same to produce both products.

However, that is what colleges do. In the College Scorecard data, the most lucrative college majors result in starting salaries that are about two and a half times greater than the college majors that result in the lowest salaries. Yet, students graduating with these degrees all end up with similar levels of debt and pay similar tuition along the way.

Why? Why would colleges charge the same for a student who can expect to make $75,000 per year upon graduation the same as one that can expect to make $30,000? Colleges are pricing solely off the supply curve and ignoring the differences in demand among subgroups of students.

I have discussed this idea with many people including some who work in higher education. I have not found even one person that supports the idea of colleges charging different tuition rates for different majors, but I also have not heard a cogent argument against it.

This idea would provide an efficiency to the labor market. If too many students chose a particular college major, resulting first year salaries will decline because there will be an excess supply of job seekers in the market. This will cause fewer future students to flock to this major and cause colleges to adjust their recruiting tactics and tuition prices. The market would provide a clear financial signal to colleges that would help them adjust their program sizes appropriately. The incentives would be in place to produce the right number of graduates from each major.

Students majoring in traditionally higher paying fields, like engineering and computer science, would end up paying more. Those in traditionally lower paying fields, like arts and human services, would pay less. All would be paying a fair amount tied to their future earning potential and the value the degree provides. You could argue that in the current system students enrolled in liberal arts are subsidizing those enrolled in engineering. Currently, because pricing isn’t in equilibrium across majors, many students are unable to attend because their preferred major will not pay off for them.

A few years back there was a proposal in Florida to have differential pricing for different majors at state institutions. However, this proposal was not letting the market determine pricing. Instead, it sought to lower the cost of STEM majors in an effort to draw more students to STEM majors. This would result in a glut of STEM graduates and lower starting salaries for these students. Counter to the current political discourse, it is the case that salaries in STEM fields have been growing at a slower rate than other college majors on average, which is the market saying that we have too many students pursuing STEM, not too few.

Differential pricing would likely be good for the colleges as it would maximize revenue and would help colleges get closer to the equilibrium price for each student. There is a reason why everyone on an airplane seems to pay a different fare – it maximizes revenue to the airline. Differential pricing is most often seen in businesses with high fixed and low marginal costs, which perfectly describes today’s traditional colleges. Differential pricing would also help colleges allocate costs more efficiently, as resources will flow to the demand.

This is a radical idea that I don’t think has ever been tried. The best argument I have heard against it is that it has the potential to limit students from poorer households to the pursuit of lower paying majors and to draw richer students to the higher paying majors, thus perpetuating a disparity. This could happen, but is more of a temporary cash flow issue that can be resolved with intelligent public policies.

Students need access to the capital necessary to get them through the college years and assurance that their resulting debt will be connected to their future earnings potential. That is where college financial aid offices and government support of higher education should place their focus. Students with ability and without financial means need temporary help getting them to a position where they have a job offer and a reasonable amount of college debt. We all have a stake in getting them to that point.

Let’s charge students a fair price that is determined by the value they receive from colleges and concentrate our public support on being sure they have a financial bridge from the moment they leave high school to when they graduate college. Linking their personal financial stake to their expected earnings is inherently fair, helps balance the labor market, and will cause colleges to provide training that is in demand by employers.

The two (or three) types of research projects every organization needs

Every once and awhile I’ll get a call from a former client or colleague who has started a new market research job. They will be in their first role as a research director or VP with a client-side organization. As they are now in a position to set their organization’s research agenda, they ask for my thoughts on how to structure their research spending. I have received calls like this about a dozen times over the years.

I advise these researchers that two types of research stand above all others, and that their initial focus should be to get them set up correctly. The first is tracking their product volume. Most organizations know how many products they are producing and shipping, but it is surprising to see how many lose track of where their products go from there. To do a good job, marketers must know how their products move through the distribution system all the way to their end consumer. So, that becomes my first recommendation: know precisely whom is buying and using your products at every step along the way, in as much detail as possible.

The second type of research I suggest is customer satisfaction research. Understanding how customers use products and measuring their satisfaction is critical. Better yet, the customer satisfaction measuring system should be prescriptive and indicate what is driving satisfaction and what is detracting from it.

Most marketing decisions can be made if these two types of research systems are well-designed. If a marketer has a handle on precisely whom is using their products and what is enhancing and detracting from their satisfaction, most of them are smart enough to make solid decisions.

When pressed for what the third type of research should be, I usually would say that qualitative research is important. I’d put in place a regular program of in-person focus groups or usability projects, and compel key decision makers to attend them. I once consulted for a consumer packaged goods client and discovered that not a single person in their marketing department had spoken directly with a consumer of their products in the past year. There is too much of a gulf between the corporate office and the real world sometimes, and qualitative research can help close that void.

Only when these three things are in place and being well-utilized would I recommend that we move forward with other types of research projects. Competitive studies, new product forecasting, advertising testing, etc. probably take up the lion’s share of most research budgets currently. They are important, but in my view should only be pursued after these first three types of research are fully implemented.

Many research departments get distracted by conducting too many projects of too many types. A focus is important. When decision makers have the basic numbers they need and are in tune with their customer base, they are in a good position to succeed, and it is market research’s role to provide this framework.

A forgotten man: rural respondents

I have attended hundreds of focus groups. These are moderated small group discussions, typically with anywhere from 4 to 12 participants. The discussions take place in a tricked-out conference room, decked with recording equipment and a one-way mirror. Researchers and clients sit behind this one-way mirror in a cushy, multi-tiered lounge. The lounge has comfortable chairs, a refrigerator with beer and wine, and an insane number of M&M’s. Experienced researchers have learned to sit as far away from the M&M’s as possible.

Focus groups are used for many purposes. Clients use them to test out new product ideas or new advertising under development. We recommend them to clients if their objectives do not seem quite ready for survey research. We also like to do focus groups after a survey research project is complete, to put some personality on our data and to have an opportunity to pursue unanswered questions.

I would estimate that at least half of all focus groups being conducted are being held in just three cities: New York, Chicago, and Los Angeles. Most of the other half are held in other major cities or in travel destinations like Las Vegas or Orlando. These city choices can have little to do with the project objectives – focus groups tend to be held near where the client’s offices are or in cities that are easy to fly to. Clients often cities simply because they want to go there.

The result is that early-stage product and advertising ideas are almost always evaluated by urban participants or by suburban participants who live near a large city. Smaller city, small town, and rural consumers aren’t an afterthought in focus group research. They aren’t thought about at all.

I’ve always been conscious of this, perhaps because I grew up in a rural town and have never lived in a major metropolitan area. The people I grew up with an knew best were not being asked to provide their opinions.

This isn’t just an issue in qualitative research, it happens with surveys and polls as well. Rural and small-town America is almost always underrepresented in market research projects.

This wasn’t a large issue for quantitative market research early on, as RDD telephone samples could effectively include rural respondents. Many years ago, I started adding questions into questionnaires that would allow me to look at the differences between urban, suburban, and rural respondents. I would often find differences, but pointing them out met with little excitement with clients who often seemed uninterested in targeting their products or marketing to a small-town audience.

Online samples do not include rural respondents as effectively as RDD telephone samples. The rural respondents that are in online sampling data bases are not necessarily representative of rural people. Weighting them upward does not magically make them representative.

In 30 years, I have not had a single client ask me to correct a sample to ensure that rural respondents are properly represented. The result is that most products and services are designed for suburbia and don’t take the specific needs of small-town folks into account.

All biases only matter if they affect what we are measuring. If rural respondents and suburban respondents feel the same way about something, this issue doesn’t matter. However, it can matter. It can matter for product research, it certainly matters to the educational market research we have conducted, and it is likely a hidden cause of some of the problems that have occurred with election polling.

How to succeed in business without planning

Crux Research is now entering its 16th year. This places us in the 1% of start-ups – as it is well-documented that most new businesses fail in the first few years. Few are lucky enough to make it to 16.

One of the reasons experts tend to say new companies fail is a lack of business planning. However, new business successes are not a consequence of an ability to plan. Success is a function of a couple of basic things:  having a product people want to buy and then delivering it really well. It doesn’t get much more basic than that, but too many times we get lost in our ambitions and ideas and aren’t forthright with ourselves on what we are good at and not so good at.

Business plans are neither necessary nor sufficient to business success. Small businesses that focus on them remind me a bit of when a company I was with put the entire staff through training on personal organization and time management. The people that embraced the training were mostly hyper-organized to begin with and the people who might have most benefited from the training were resistant. Small businesses that rely heavily on business planning probably are using it more as reflection of their natural organizational tendencies than as a true guide to running a business. There are, of course, exceptions.

Entrepreneurs that use business plans seem to fall into two categories. First, they may feel constrained by them. If they have investors, they feel a pressure to stay within the confines of the plan even when the market starts telling them to pivot to a new strategy. They start to think adherence to the plan is, in itself, a definition of success. In short, they tend to be “process” people as opposed to “results” people.  Entrepreneurs that are “results” people are the successful ones. If you are a “process” person I’d suggest that you may be a better fit for a larger company.

Second, and this is more common, many entrepreneurs spend a lot of time in business planning before launching their company and then promptly file the plan away and never look at it again.

That is how it went for me in the corporate world. We would craft annual plans each spring and then never look at them again until late in the year when it was time to create the plan for the following year. It was sort of like giving blueprints to a contractor and then having them build whatever they want to anyway but then giving them a hard time for not following the plan… even if they had built a masterpiece.

We’d also create five-year plans every year. That never made a lot of sense to me as it was an acknowledgement that would couldn’t really plan more than a year out.

I am not against business planning in concept. I agree with Winston Churchill who said “plans are of little importance, but planning is essential.” The true value of most strategic plans is found in the thought process that is gone through to establish them and not in the resulting plan.

I’d suggest that anyone who starts a company list out the top reasons why your company might fail and then think through how you are going to prevent these things from happening. If the reasons for your potential failure are mostly under your control, you’ll be okay. We did this. We identified that if we were going to fail it was going to most likely result from an inability to generate sales, as this wasn’t where our interest or core competence lay. So, we commissioned a company to develop leads for us, had a plan for reaching out to past clients, and planned to hire a salesperson (something that never happened). The point is, while we didn’t really have a business plan, we did think through how to prevent a problem that would prevent our success.  

Before launching Crux Research, we did lay out some key opportunities and threats and thought hard about how to address what we saw as the barriers to our success. Late each fall we think about goals for the coming year, how we can improve what we are doing, and where we can best spend our time in marketing and sales. That sort of thinking is important and it is good to have formalized. But we’ve never had a formal business plan. Maybe we have succeeded in spite of that, but I tend to think we have succeeded because of it. 

Critics would probably say that the issue here is business plans just need to be better and more effective. I don’t think that is the case. The very concept of business planning for a small business can be misguided. It is important to be disciplined as an entrepreneur/small business owner. A focus is paramount to success. But I don’t think it is all that important to be “planned.” You miss too many opportunities that way.

Again, our company has never had a financial plan. I attribute much of our success to that. The company has gone in directions we would never have been able to predict in advance. We try to be opportunistic and open-minded and to stay within the confines of what we know how to do. We prefer having a high level of self-awareness of what we are good and not so good at and to create and respond to opportunities keeping that in mind. As Mike Tyson once said “everyone has a plan until they get punched in the mouth.” I wonder how many small businesses have plans that quickly got punched.

This has led us to many crossroads where we have to choose whether to expand our company or to refuse work. We’ve almost always come down on the side of staying small and being selective in the work we take on.

In the future we are likely to become even more selective in who we take on as clients. We’ve always said that we want a few core things in a client. Projects have to be financially viable, but more importantly they have to be interesting and a situation that will benefit from our experience and insight.  We have developed a client base of really great researchers who double as really great people. It sums up to a business that is a lot of fun to be in.

I would never discourage an entrepreneur from creating a business plan, but I’d advise them to think hard about why they are doing so. You’ll have no choice if you are seeking funding, as no investor is going to give money to a business that doesn’t have a plan. If you are in a low margin business, you have to have a tight understanding of your cash flow and planning that out in advance is important. I’d suggest you keep any plans broad and not detailed, conduct an honest assessment of what you are not so good at as well as good at, and prepare to respond to opportunities you can’t possibly foresee. Don’t judge your success by your adherence to the plan and if your funders appear too insistent on keeping to the plan, find new funders.

Wow! Market research presentations have changed.

I recently led an end-of-project presentation over Zoom. During it, I couldn’t help but think how market research presentations have changed over the years. There was no single event or time period that changed the nature of research presentations, but if you teleported a researcher from the 1990’s to a modern presentation they would feel a bit uncomfortable.

I have been in hundreds of market research presentations — some led by me, some led by others, and I’ve racked up quite a few air miles getting to them. In many ways, today’s presentations are more effective than those in the past. In some other ways, quality has been lost. Below is a summary of some key differences.

Today’s presentations are:

  • Far more likely to be conducted remotely over video or audio. COVID-19 disruptions acted as an accelerant onto this trend which was happening well before 2020. This has made presentations easier to schedule because not everyone has to be available in the office. This allows clients and suppliers to take part from their homes, hotels, and even their vehicles. It seems clear that a lasting effect of the pandemic will be that research presentations will be conducted via Zoom by default. There are plusses and minuses to this. For the first time in 30 years, I find myself working with clients whom I have never met in-person.
  • Much more likely to be bringing in data and perspectives from outside the immediate project. Research projects and presentations tended to be standalone events in the past, concentrating solely on the area of inquiry the study addressed. Today’s presentations are often integrated into a wider reaching strategic discussion that goes beyond the questions the research addresses.
  • More interactive. In yesteryear, the presentation typically consisted of the supplier running through the project results and implications for 45 minutes followed by a period of Q&A. It was rare to be interrupted before the Q&A portion of the meeting. Today’s presentations are often not presentations at all. As a supplier we feel like we are more like emcee’s leading a discussion than experts presenting findings.
  • More inclusive of upper management. We used to present almost exclusively to researchers and mid-level marketers. Now, we tend to see a lot more marketing VP’s and CMOs, strategy officers, and even the CEO on occasion. It used to be rare that our reports would make it to the CEOs desk. Now, I’d say most of the time they do. This is indicative of the increasing role data and research has in business today.
  • Far more likely to integrate the client’s perspective. In the past, internal research staff rarely tried to change or influence our reports and presentations, preferring to keep some distance and then separately add their perspective. Clients have become much more active in reviewing and revising supplier reports and presentations.

Presentations from the 1990’s were:

  • A more thorough presentation of the findings of the study. They told a richer, more nuanced story. They focused a lot more on storytelling and building a case for the recommendations. Today’s presentations often feel like a race to get to the conclusions before you get interrupted.
  • More confrontational. Being challenged on the study method, data quality, and interpretations was more commonplace a few decades ago. I felt a much greater need to prepare and rehearse than I do today because I am not as in control of the flow of the meetings as I was previously. In the past I felt like I had to know the data in great detail, and it was difficult for me to present a project if I wasn’t the lead analyst on it. Today, that is much less of a concern.
  • More strategic. This refers more to the content of the studies than the presentation itself. Since far fewer studies were being done, the ones that were tended to be informing high consequence decisions. While plenty of strategic studies are still conducted, there are so many studies being done today that many of them are informing smaller, low-consequence, tactical decisions.
  • More relaxed. Timelines were more relaxed and as a result research projects were planned well in advance and the projects fed into a wider strategic process. That still happens, but a lot of today’s projects are completed quickly (often too quickly) because information is needed to make a decision that wasn’t even on the radar a few weeks prior.
  • More of a “show.” In the past we rehearsed more, were concerned about the graphical design of the slides, and worried about the layout of the room. Today, there is rarely time for that.
  • More social. Traveling in for a presentation meant spending time beforehand with clients, touring offices, and almost always going to lunch or dinner afterword. Even before the COVID/Zoom era, more recent presentations tended to be “in and out” affairs – where suppliers greet the clients, give a presentation, and leave. While there are many plusses to this, some (I’d actually say most) of the best researchers I know are introverts who were never comfortable with this forced socialization. Those types of people are going to thrive in the new presentation environment.

Client-side researchers were much more planned out in the past. Annually, they would go through a planning phase where all the projects for the year would be budgeted and placed in a timeline. The research department would then execute against that plan. More recently, our clients seem like they don’t really know what projects they will be working on in a few weeks’ time – because many of today’s projects take just days from conception to execution.

I have also noticed that while clients are commissioning more projects they seem to be using fewer suppliers than in the past. I think this is because studies are being done so quickly they don’t have time to manage more than a few supplier relationships. Bids aren’t as competitive and are more likely to be sole-sourced.

Clients are thus developing closer professional relationships with their suppliers. Suppliers are closer partners with clients than ever before, but with this comes a caution. It becomes easy to lose a third-party objectivity when we get too close to the people and issues at hand and when clients have too heavy a hand in the report process. In this sense, I prefer the old days, where we provided a perspective and our clients would then add a POV. Now, we often meld the two into one presentation, and at time we lose the value that comes from a back and forth disagreement over what the findings mean to a business.

If I teleported my 1990’s self to today I would be amazed at how quickly projects go from conception to final presentation. Literally, this happens in about one-third the time it used to. There are many downsides of going too fast and clients rarely focus or care about them. While there are dangers to going too fast, clients seem to prefer getting something 90% right and getting it done tomorrow, than waiting for a perfect project.

There is even a new category of market research called “agile research” that seeks to provide real-time data. I am sure it is a category that will grow, but those employing it need to keep in mind that providing data faster than managers can act on it can actually be a disservice to the client. It is an irony of our field that more data and continuous data can actually slow down decision making.  

Today’s presentations are less stressful, more inclusive, and more strategic. The downside is there are probably too many of them – clients are conducting too many projects on minor issues, they don’t always learn thoroughly from one study before moving onto the next, and researchers are sometimes being rewarded more for getting things done than for providing insight into the business.


Visit the Crux Research Website www.cruxresearch.com

Enter your email address to follow this blog and receive notifications of new posts by email.