Researchers should be mindful of “regression toward the mean”

There is a concept in statistics known as regression toward the mean that is important for researchers to consider as we look at how the COVID-19 pandemic might change future consumer behavior. This concept is as challenging to understand as it is interesting.

Regression toward the mean implies that an extreme example in a data set tends to be followed by an example that is less extreme and closer to the “average” value of the population. A common example is if two parents that are above average in height have a child, that child is demonstrably more likely to be closer to average height than the “extreme” height of their parents.

This is an important concept to keep in mind in the design of experiments and when analyzing market research data. I did a study once where we interviewed the “best” customers of a quick service restaurant, defined as those that had visited the restaurant 10 or more times in the past month. We gave each of them a coupon and interviewed them a month later to determine the effect of the coupon. We found that they actually went to the restaurant less often the month after receiving the coupon than the month before.

It would have been easy to conclude that the coupon caused customers to visit less frequently and that there was something wrong with it (which is what we initially thought). What really happened was a regression toward the mean. Surveying customers who had visited a large number of times in one month made it likely that these same customers would visit a more “average” amount in a following month whether they had a coupon or not. This was a poor research design because we couldn’t really assess the impact of the coupon which was our goal.

Personally, I’ve always had a hard time understanding and explaining regression toward the mean because the concept seems to be counter to another concept known as “independent trials”. You have a 50% chance of flipping a fair coin and having it come up heads regardless of what has happened in previous flips. You can’t guess whether the roulette wheel will come up red or black based on what has happened in previous spins. So, why would we expect a restaurant’s best customers to visit less in the future?

This happens when we begin with a skewed population. The most frequent customers are not “average” and have room to regress toward the mean in the future. Had we surveyed all customers across the full range of patronage there would be no mean to regress to and we could have done a better job of isolating the effect of the coupon.

Here is another example of regression toward the mean. Suppose the Buffalo Bills quarterback, Josh Allen, has a monster game when they play the New England Patriots. Allen, who has been averaging about 220 yards passing per game in his career goes off and burns the Patriots for 450 yards. After we are done celebrating and breaking tables in western NY, what would be our best prediction for the yards Allen will throw for the second time the Bills play the Patriots?

Well, you could say the best prediction is 450 yards as that is what he did the first time. But, regression toward the mean would imply that he’s more likely to throw close to his historic average of 220 yards the second time around. So, when he throws for 220 yards the second game it is important to not give undue credit to Bill Belichick for figuring out how to stop Allen.

Here is another sports example. I have played (poorly) in a fantasy baseball league for almost 30 years. In 2004, Derek Jeter entered the season as a career .317 hitter. After the first 100 games or so he was hitting under .200. The person in my league that owned him was frustrated so I traded for him. Jeter went on to hit well over .300 the rest of the season. This was predictable because there wasn’t any underlying reason (like injury) for his slump. His underlying average was much better than his current performance and because of the concept of regression toward the mean it was likely he would have a great second half of the season, which he did.

There are interesting HR examples of regression toward the mean. Say you have an employee that does a stellar job on an assignment – over and above what she normally does. You praise her and give her a bonus. Then, you notice that on the next assignment she doesn’t perform on the same level. It would be easy to conclude that the praise and bonus caused the poor performance when in reality her performance was just regressing back toward the mean. I know sales managers who have had this exact problem – they reward their highest performers with elaborate bonuses and trips and then notice that the following year they don’t perform as well. They then conclude that their incentives aren’t working.

The concept is hard at work in other settings. Mutual funds that outperform the market tend to fall back in line the next year. You tend to feel better the day after you go to the doctor. Companies profiled in “Good to Great” tend to have hard times later on.

Regression toward the mean is important to consider when designing sampling plans. If you are sampling an extreme portion of a population it can be a relevant consideration. Sample size is also important. When you have just a few cases of something, mathematically an extreme response can skew your mean.

The issue to be wary of is that when we fail to consider regression toward the mean, we tend to overstate the importance of correlation between two things. We think our mutual fund manager is a genius when he just got lucky, that our coupon isn’t working, or that Josh Allen is becoming the next Drew Brees. All of these could be true, but be careful in how you interpret data that result from extreme or small sample sizes.

How does this relate to COVID? Well, at the moment, I’d say we are still in an “inflated expectations” portion of a hype curve when we think of what permanent changes may take place resulting from the pandemic. There are a lot of examples. We hear that commercial real estate is dead because businesses will keep employees working from home. Higher education will move entirely online. In-person qualitative market research will never happen again. Business travel is gone forever. We will never again work in an office setting. Shaking hands is a thing of the past.

I’m not saying there won’t be a new normal that results from COVID, but if we believe in regression toward the mean and the hype curve we’d predict that the future will look more like the past than how it is currently being portrayed. The post-COVID world will certainly look more like the past than a more extreme version of the present. We will naturally regress back toward the past and not to a more extreme version of current behaviors. The “mean” being regressed to has likely changed, but not as much as the current, extreme situation implies.

“Margin of error” sort of explained (+/-5%)

It is now September of an election year. Get ready for a two-month deluge of polls and commentary on them. One thing you can count on is reporters and pundits misinterpreting the meaning behind “margin of error.” This post is meant to simplify the concept.

Margin of error refers to sampling error and is present on every poll or market research survey. It can be mathematically calculated. All polls seek to figure out what everybody thinks by asking a small sample of people. There is always some degree of error in this.

The formula for margin of error is fairly simple and depends mostly on two things: how many people are surveyed and their variability of response. The more people you interview, the lower (better) the margin of error. The more the people you interview give the same response (lower variability), the better the margin of error. If a poll interviews a lot of people and they all seem to be saying the same thing, the margin of error of the poll is low. If the poll interviews a small number of people and they disagree a lot, the margin of error is high.

Most reporters understand that a poll with a lot of respondents is better than one with fewer respondents. But most don’t understand the variability component.

There is another assumption used in the calculation for sampling error as well: the confidence level desired. Almost every pollster will use a 95% confidence level, so for this explanation we don’t have to worry too much about that.

What does it mean to be within the margin of error on a poll? It simply means that the two percentages being compared can be deemed different from one another with 95% confidence. Put another way, if the poll was repeated a zillion times, we’d expect that at least 19 out of 20 times the two numbers would be different.

If Biden is leading Trump in a poll by 8 points and the margin of error is 5 points, we can be confident he is really ahead because this lead is outside the margin of error. Not perfectly confident, but more than 95% confident.

Here is where reporters and pundits mess it up.  Say they are reporting on a poll with a 5-point margin of error and Biden is leading Trump by 4 points. Because this lead is within the margin of error, they will often call it a “statistical dead heat” or say something that implies that the race is tied.

Neither is true. The only way for a poll to have a statistical dead heat is for the exact same number of people to choose each candidate. In this example the race isn’t tied at all, we just have a less than 95% confidence that Biden is leading. In this example, we might be 90% sure that Biden is leading Trump. So, why would anyone call that a statistical dead heat? It would be way better to be reporting the level of confidence that we have that Biden is winning, or the p-value of the result. I have never seen a reporter do that, but some of the election prediction websites do.

Pollsters themselves will misinterpret the concept. They will deem their poll “accurate” as long as the election result is within the margin of error. In close elections this isn’t helpful, as what really matters is making a correct prediction of what will happen.

Most of the 2016 final polls were accurate if you define being accurate as coming within the margin of error. But, since almost all of them predicted the wrong winner, I don’t think we will see future textbooks holding 2016 out there as a zenith of polling accuracy.

Another mistake reporters (and researchers make) is not recognizing that the margin of error only refers to sampling error which is just one of many errors that can occur on a poll. The poor performance of the 2016 presidential polls really had nothing to do with sampling error at all.

I’ve always questioned why there is so much emphasis on sampling error for a couple of reasons. First, the calculation of sampling error assumes you are working with a random sample which in today’s polling world is almost never the case. Second, there are many other types of errors in survey research that are likely more relevant to a poll’s accuracy than sampling error. The focus on sampling error is driven largely because it is the easiest error to mathematically calculate. Margin of error is useful to consider, but needs to be put in context of all the other types of errors that can happen in a poll.

Online education will need to change before it rules higher education

We recently conducted a poll of college students around the world about their experiences with online education this spring that resulted from the pandemic. The short answer is students didn’t fare well and are highly critical of the ability of online education to engage them and to deliver instruction. This isn’t a subtle, nuanced finding. A large majority of college students worldwide thought the online education they received this spring was ineffective and unengaging.

I held out hope that the pandemic would be the event that finally kickstarted online education. Our poll results have me doubting it will, which is a shame as online education holds enormous potential. It is a new technology that is, for some reason, being held back. If you think about it, we have had all the technology needed to take education online for at least 10 years, yet for the most part the traditional university system has remained as it was a generation ago.

I’ve always been interested in new “media” technologies because I’ve noticed a pattern in their emergence. Almost always, they begin as a nifty new delivery system for content that was developed with the “old” media. The earliest radio shows largely consisted of people reading the newspapers aloud and playing music. Early television mostly adapted content from radio – serialized dramas, variety shows, baseball games, etc. The Internet 1.0 largely just electronically expectorated content that existed in other forms.

After a bit of a gestation period, “new” media eventually thrive as they take advantage of their technological uniqueness and content evolves along with the new distribution system. The result is something really special and not just a new way to deliver old things.

There are many examples. Radio moved to become central to family entertainment and ritual in a way the newspaper could not. Television developed the Saturday morning lineup, the situation comedy, talk shows, etc., none of which could have worked as well on radio. And, the Internet evolved and became interactive, with user-created content, product reviews, with a melding of content and commerce that isn’t possible in other media. In all cases, the “new” media gestated awhile by mimicking the old but once they found their way their value grew exponentially. The old media didn’t go away, but got repositioned to a narrower niche.

This hasn’t happened in higher education. Streaming your lecture on Zoom might be necessary during a pandemic but it is not what online education should be about. Students consistently tell us it doesn’t work for them. Parents and students don’t feel it provides the value they expect from college, which is why we are starting to see lawsuits where students are demanding tuition refunds from colleges that moved education online this spring.

We composed a post a little while ago that posited that the reason digital textbooks really haven’t made much of a difference in colleges is because textbook publishers have prevented this synergy from happening. Most digital textbooks today are simply a regurgitation of a printed textbook that you can read on a computer. Our surveys show that the number one way a digital textbook is read remains by viewing a PDF. That is hardly taking advantage of what today’s technology has to offer.

The potential for the digital textbook is much greater. In fact, it wouldn’t be a textbook at all. Instead, there could be a digital nexus of all that is going on in a course, conducted, coached, and curated by the instructor. Imagine a “book” that could take you on an interactive tour. It could link you to lectures by world-renown people. It could show practitioners applying the knowledge they gained in the course. It could contain formative assessments where you could determine how you are progressing and then adapt to focus you where you need individualized help. A tutor would be a link away. Other students could comment and help you.

Your instructor could become a coach rather than a sage. This wouldn’t be a textbook at all, but a melding of course materials and instruction and collaborative tools.

This technology exists today, yet publishers and colleges have too much of a self-interest not to innovate. Education is suffering because of it.

This spring most college instructors had one or two weeks to figure out how to move their instruction online with little help from textbook publishers or technology companies. They had no choice but to adapt their existing course to a new delivery system. So, they pointed a camera on themselves and called it online education.

It is no wonder that online education largely failed our students. Every poll I have seen, including a few Crux has conducted, has shown that students found online education to be vastly inferior to traditional instruction this spring.

But, did you know this isn’t new? College students have long been critical of online education. I’ve asked questions about online education to college students for almost 20 years. While many appreciate the convenience of an online course and that it can cost less, a very large majority of those taking online courses say they aren’t an effective way to learn. Almost all say that they would have learned better in a traditional course. It is a rare student that chooses an online course because it is an effective way to learn. When they choose an online course, it is because it fits better into their life situation and not because it is an effective way to learn.

Why? Because online course providers really haven’t taken advantage of a “new” medium. They are still adapting traditional education and placing it online rather than embracing the uniqueness that online education can provide. They are firmly ensconced in Internet 1.0 a decade or two after all other industries have moved on. Compared to a decade ago we shop completely differently. We watch entertainment completely differently. We communicate with others completely differently. Yet, our children attend college the same way their parents and grandparents did.

Course management systems do exist, but to date they haven’t fundamentally changed the nature of a college course. We ask about course management systems on surveys as well, and college students find them to be moderately helpful, but hardly game changing.

One of Crux’s largest clients is a supplemental education company that provides resources to college students who don’t feel they are getting the support they need from their college or their professors. This company has been one of the best performing companies in the US since COVID-19 hit and so many courses moved online. This client is well-managed and has a great vision and brilliant employees. But, if educators had fully figured out how to effectively educate online, I don’t think they could be as successful as they have been because students wouldn’t have such a pressing need for outside help. Because of higher education’s unwillingness or inability to adapt, I expect this client to thrive for a long time.

It is sad in a way to think that our colleges and universities, who should be on the forefront of technology and innovation, are sadly lacking in adapting course materials and instruction to the Internet. Especially when you consider that these are the same entities that largely invented the Internet.

Living near Rochester NY, it is easy to see a parallel to the Eastman Kodak company. Kodak had one of the strongest brands in the world, was tightly identified with imaging and photography, and had invented almost all of the core technologies needed for digital photography. All at a time when the number of images consumers were about to capture was about to explode literally by a factor of about 10,000, maybe 100,000. But, because of an inability to break out of an old way of thinking and an inertia to hang on too long to an “old” media, one of America’s great companies was essentially reduced to a business school case in how to grab defeat from the jaws of opportunity.

Is this a cautionary tale for colleges and universities? Sure. I suspect that elite college brands will continue to do well as they cater to a wealthy demographic that has done quite well during the pandemic. But, for the rest of us, who send students to non-elite institutions, I expect to see colleges face enormous financial pressures and to see many college brands go the road of Kodak over the next decade. Their ticket to a better path is to more effectively use technology.

Online education has the potential to cure some of what ails the US higher education system. It can adapt quickly to market demand for workers. It can provide much wider access to the best and brightest teachers. It can aggregate a mass of students who might be interested in a highly specialized field, and thus become more targeted. And, it may finally be what finally fixes the high cost of higher education.

Will online education will thrive in the US? Not until it changes to take advantage of what an interconnected world has to offer. The time is right for colleges to truly tap into the power of what online education can be. This is really the only way colleges will be able to charge the tuition levels they have become accustomed to charging and until online education becomes synonymous with quality education, many colleges will struggle.

This is taking far too long but I am hopeful that kickstarting this process will be one silver lining to come out of the upheaval to education that has been caused by the pandemic.

I have more LinkedIn contacts named “Steve” than contacts who are Black

There have been increasing calls for inclusiveness and fairness across America and the world. The issues presented by the MeToo and Black Lives Matter movements affect all sectors of society and the business world. Market research is no exception. Recent events have spurred me to reflect on my experiences and to think about whether the market research field is diverse enough and ready to make meaningful changes. Does market research have structural, systemic barriers preventing women and minorities from succeeding?

My recollections are anecdotal – just one person’s experiences when working in market research for more than 30 years. What follows isn’t based on an industry study or necessarily representative of all researchers’ experiences.

Women in Market Research

When it comes to gender equity in the market research field, my gut reaction is to think that research is a good field for women and one that I would recommend. I reviewed Crux Research’s client base and client contacts. In 15 years, we have worked with about 150 individual research clients across 70 organizations. 110 (73%) of those 150 clients are female. This dovetails with my recollection of my time at a major research supplier. Most of my direct clients there were women.

Crux’s client base is largely mid-career professionals – I’d say our typical client is a research manager or director in his/her 30’s or 40’s. I’d conclude that in my experience, women are well represented at this level.

But, when I look through our list of 70 clients and catalog who the “top” research manager is at these organizations, I find that 42 (60%) of the 70 research VPs and directors are male. And, when I catalog who these research VP’s report into, typically a CMO, I find that 60 (86%) of the 70 individuals are male. To recap, among our client base, 73% of the research managers are female, 40% of the research VPs are female, and 14% of the CMO’s are female.

This meshes with my experience working at a large supplier. While I was there, women were well-represented in our research director and VP roles but there were almost no women represented in the C-suite or among those that report to them. There seem to be a clear but firm glass ceiling in place in market research suppliers and in clients.

Minorities in Market Research

My experience paints a bleaker picture when I think of ethnic minority representation in market research. Of our 150 individual research clients, just 25 (17%) have been non-white and just 3 (2%) have been black. Moving up the corporate ladder, in only 5 (13%) of our 70 clients is the top researcher in the organization non-white and in only 4 (6%) of the 70 companies is the CMO non-white, and none of the CMOs are black. Undoubtedly, we have a long way to go.

A lack of staff diversity in research suppliers and market research corporate staffs is a problem worth resolving for a very important reason: market researchers and pollsters are the folks providing the information to the rest of the world on diversity issues. Our field can’t possibly provide an appropriate perspective to decision makers if we aren’t more diverse. Our lack of diversity affects the conversation because we provide the data the conversation is based upon.  

Non-profits seem to be a notable exception when it comes to ethnic diversity. I have had large non-profit clients that have wonderfully diverse employee bases, to the point where it is not uncommon to attend meetings and Zoom calls where I am the only white male in the session. These non-profits make an effort to recruit and train diverse staffs and their work benefits greatly from the diversity of perspectives this brings. There is a palpable openness of ideas in these organizations. Research clients and suppliers would do well to learn from their example.  

I can’t think of explicit structural barriers that limit the progression of minorities thought the market research ranks, but that just illustrates the problem: the barriers aren’t explicit, they are more subtle and implicit. Which is what makes them so intractable.

We have to make a commitment to develop more diverse employee bases. I worked directly for the CEO of a major supplier for a number of years. One thing I respected about him was he was confident enough in himself that he was not afraid to hire people who were smarter than him or didn’t think like him or came from an entirely different background. It made him unique. In my experience, most hiring managers unintentionally hire “mini-me’s” – younger variants of themselves whom they naturally like in a job interview. Well, if the hiring managers are mostly white males and they are predisposed to hire a lot of “mini-me’s” over time this perpetuates a privilege and is an example of an unintentional, but nonetheless structural bias that limits the progress of women and minorities.

If you don’t think managers tend to hire in their own image, consider a recent Economist article that states “In 2018 there were more men called Steve than there were women among the chief executives of FTSE 100 companies.” I wouldn’t be surprised if there are more market researchers in the US named Steve than there are black market researchers.

To further illustrate that we naturally seek people like ourselves, I reviewed my own LinkedIn contact list. This list is made up of former colleagues, clients, people I have met along the way, etc. It is a good representation of the professional circle I exist within. It turns out that my LinkedIn contact list is 60% female and has 25% non-whites. But, just 3% of my LinkedIn contacts are black. And, yes, I have more LinkedIn contacts named Steve than I have contacts who are black.

This is a problem because as researchers we need to do our best to cast aside our biases and provide an objective analysis of the data we collect. We cannot do that well if we do not have a diverse array of people working on our projects.

Many managers will tell you that they would like to hire a minority for a position but they just don’t get quality candidates applying. This is not taking ownership of the issue. What are you doing to generate candidates in the first place?

It is all too easy to point the finger backwards at colleges and universities and say that we aren’t getting enough qualified candidates of color. And that might be true. MBA programs continue to enroll many more men than women and many more whites than non-whites. They should be taken to task for this. As employers we also need to be making more demands on them to recruit women and minorities to their programs in the first place.

I like that many research firms have come out with supportive statements and financial contributions to relevant causes recently. This is just a first step and needs to be the catalyst to more long-lasting cultural changes in organizations.

We need to share best practices, and our industry associations need to step up and lead this process. Let’s establish relationships with HCBU’s and other institutions to train the next generation of black researchers.

The need to be diverse is also important in the studies we conduct. We need to call more attention to similarities and differences in our analyses – and sample enough minorities in the first place so that we can do this. Most researchers do this already when we have a reason to believe before we launch the study that there might be important differences by race/ethnicity. However, we need to do this more as a matter of course, and become more attuned to highlighting the nuances in our data sets that are driven by race.

Our sample suppliers need to do a better job of recruiting minorities to our studies, and to ensure that the minorities we sample are representative of a wider population. As their clients, we as suppliers need to make more demands about the quality of the minority samples we seek.

We need an advocacy group for minorities in market research. There is an excellent group, Women in Research https://www.womeninresearch.org/ advocating for women. We need an analogous organization for minorities.

Since I am in research, I naturally think that measurement is key to the solution. I’ve long thought that organizations only change what they can measure. Does your organization’s management team have a formal reporting process that informs them of the diversity of their staff, of their new hires, of the candidates they bring in for interviews? If they do not, your organization is not poised to fix the problem. If your head of HR cannot readily tell you what proportion of your staff is made up of minorities, your firm is likely not paying enough attention.

Researchers will need to realize that their organizations will become better and more profitable when they recruit and develop a more diverse client base. Even though it is the right thing to do, we need to view resolving these issues not solely as altruism. It is in our own self-interest to work on this problem. It is truly the case that if we aren’t part of the solution, we are likely part of the problem. And again, because we are the ones that inform everyone else about public opinion on these issues, we need to lead the way.

My belief is it that this issue will be resolved by Millennials once they get to an age when they are more senior in organizations. Millennials are a generation that is intolerant to unfairness of this sort and notices the subtle biases that add up. They are the most diverse generation in US history. The oldest Millennials are currently in their mid-30’s. In 10-20 years’ time they will be in powerful positions in business, non-profits, education, and government.

Optimistically, I believe Millennials will make a big difference. Pessimistically, I wonder if real change will happen before they are the ones managing suppliers and clients, as thus far the older generations have not shown that they are up to the task.

Shift the Infield, Go for Two, and Pull the Goalie Sooner!

Moneyball is one of my favorite books. It combines many interests of mine – statistics, baseball, and management. I once used it to inspire a client to think about their business differently. This client was a newly-named President of a firm and had brought us in to conduct some consumer market research. New management teams often like to bring in new research suppliers and shed their old ones, and in this case we were the beneficiaries.

In our initial meeting, I asked some basic marketing questions about how they decide to price their products or how much to spend on advertising. Each time his response was “this is how we have always done it” rather than a well-thought out rationale supporting his decision. For instance, most of his products were priced to retailers at 50% of the price to consumers because that is how it had been for decades. I asked him “what are the odds that your optimal pricing is 50% rather than something higher or lower?” What are the chances that a round number like 50% could be optimal for all products in all cases when he literally had thousands of products?

I sent him a copy of Moneyball when I returned from the trip because I knew he was a sports fan. He read it immediately. It sparked him to commission a consulting firm to delve deeply into pricing models and ultimately led to a significant change in their pricing policies. They no longer used 50% as a target and established different wholesale prices for each of their SKU’s based on demand and updated these prices regularly. A few years later, he told me that decision literally saved his firm millions of dollars and the pricing efficiency helped to distribute his products more effectively. He said this was probably the project he had led that had the biggest impact on his business since he had been there.

Businesses can use sports analogies too readily, but in this case it really worked. The rise of statisticians in sports has worked and there are lessons that businesses can learn from this.

I find it fascinating when old-timers and sports talk radio hosts lament the rise of “analytics” in sports. You can see the impact of statisticians every time you see a baseball team set up in a defensive shift, when you see a football team go for it on fourth down, or when you see a hockey team pull its goalie earlier than usual. These decisions being made more frequently and in situations where the prior norms of the game would have prevented them from happening. It is all because data jockeys have been given a seat at the sports management table to the chagrin of the purists.

But data geeks haven’t totally taken over sports and longstanding traditions continue to hold sway. For instance, in baseball it can be shown that more runs on average are scored in the first inning than in any other inning. This makes sense, as the first inning is the only time in the game when you can be sure your best hitters will be at the top of the batting order. So, why don’t major league teams start their closer and have him pitch the first inning? Instead, they reserve their most powerful pitcher for the 9th inning, when, more often or not, the game is already decided. I’ve been predicting that teams will figure this out and start their closer for about 20 years now and they haven’t done it yet. (The Tampa Rays did something close to this and had an “opener” pitcher in their rotation, but it didn’t work well because this pitcher wasn’t their most powerful arm.)

Similarly, hockey teams continue to be slow to pull their goalie when behind late in the game. Hockey coaches also continue to make a decision that baffles me every time. They are down by one goal late in the game so they pull their goalie and promptly surrender a goal. The first thing they do is put their goalie back in which makes no rational sense at all. If you are willing to take the risk of being scored upon when losing by one goal, you should be even more willing to do so when losing by two goals. There is an excellent paper on pulling the goalie (“Pulling the Goalie:  Hockey and Investment Implications.”) which shows that coaches aren’t pulling their goalie even close to quick enough.

These sports cases are interesting because it is the fans that always seem to notice the coaching strategy errors before the coaches and general managers. This illustrates the value of an outside perspective in organizations that have longstanding policies and traditions. I don’t think my client could have accomplished his pricing changes if he wasn’t brand new to the organization or if he didn’t hire a consulting firm to work out the optimal strategy. This change was not going to come from within his organization.

Businesses have been slow to adapt their thinking despite the vast amount of data at their disposal. Decisions are made all the time without consulting what the data are indicating. More relevant to our industry, in most organizations market research is still seen as a support function to marketing, as opposed to its equal. I don’t think I have ever heard of an organization where market research reports directly to senior management or where marketing reports into research, yet we often hear senior managers say that connecting to customers is the most critical part of their organization’s success.

Many saw Moneyball as a book about sports or a great movie. I saw it as one of the most important business books ever written. Its key message is to use data to break out of existing decision patterns, often to great success.

What are the best COVID-19 polls?

The COVID-19 crisis is affecting all types of organizations. Many, including some of our clients, are commissioning private polls to help predict the specific impact of the pandemic on their business. Fortunately, there are a number of well-regarded research and polling organizations conducting polls that are publicly released. Unfortunately, there are also disreputable polls out there and can be challenging to sort out the good from the bad.

We’ve been closely watching the COVID-19 polls and have found some that stand out from the rest. We felt it would be a good idea to list them here to save you some time as you look for polling information.

  • First, although it is not a poll, a useful site to look at is the Institute for Health Metrics and Evaluation at the University of Washington. This site contains the results of a model projecting numbers of deaths from COVID-19, beds needed versus hospital capacity, etc. This is one of the most credible models out there, and the one that seems to be cited the most in the media and by the federal government.
  • Johns Hopkins University maintains a coronavirus tracking center which is the definitive place to go to track cases, hospitalizations, and deaths from COVID-19. 

Below is a list of opinion polls we’ve found most interesting. There are a lot of polls out there. The ones listed below are from trusted organizations and would be a good place to start your search. There are many polls available that concentrate on things like the public’s opinion of the government’s handling of the crisis. The polls below go a bit deeper and are far more interesting in our view. There are likely other good polls out there, but these are the best ones we have found thus far.

  • The Harris Poll COVID-19 trackerThis is perhaps the most comprehensive COVID-19 polling we have discovered, and it tracks back to early March. If you have time for just one polling site this is probably the one to check out.
  • PewPew is a widely-respected organization that has conducted many polls on COVID-19 topics.
  • The COVID Impact Survey. This is an independent, non-governmental survey being conducted by NORC along with some respected foundations. 
  • Dynata. Dynata has a tracking poll going on COVID-19 that is interesting because it spans multiple countries. Dynata is also doing a “symptom map” based on their polling worldwide. This is interesting as it shows how symptoms are trending around the world, in the US by state, and even in NYC by neighborhood. However, we feel that a Google Trends search would provide better data that survey research on symptoms. 
  • IPSOS.  IPSOS is also conducting worldwide polls
  • Simpson Scarborough. This poll is specific to higher education and the implications of COVID-19 on college students. If you work in higher education or have a college-aged child, you are likely to find this one interesting.
  • University of Massachusetts Amherst. This one is different and interesting. It shows the results of an ongoing survey of infectious disease experts, containing their predictions for the impacts of the disease.  It is updated weekly.  FiveThirtyEight is summarizing this work and it is probably easiest to read their summaries than to go to the original source. I must say, though, I have been watching this poll carefully, and the experts haven’t been all that accurate in their predictions, missing on the high side consistently.

There are undoubtedly many more good polls out there. Those mentioned above are from non-partisan, trusted organizations.

How COVID-19 may change Market Research

Business life is changing as COVID-19 spreads in the US and the world. In the market research and insights field there will be both short-term and long-term effects. It is important that clients and suppliers begin preparing for them.

This has been a challenging post to write. First, in the context of what many people are going though in their personal and business lives as a result of this disruption, writing about what might happen to one small sector of the business world can come across as uncaring and tone-deaf, which is not the intention. Second, this is a quickly changing situation and this post has been rewritten a number of times in the past week. I have a feeling it may not age well.

Nonetheless, market research will be highly impacted by this situation. Below are some things we think will likely happen to the market research industry.

  • An upcoming recession will hit the MR industry hard. Market research is not an investment that typically pays off quickly. Companies that are forced to pare back will cut their research spending and likely their staffs.
  • Cuts will affect clients more than suppliers. In previous recessions, clients have cut MR staff and outsourced work to suppliers. This is an opportunity for suppliers that know their clients’ businesses well and can step up to help.
  • Unlike a lot of other types of industries, it is the large suppliers that are most at risk of losing work. Publicly-held research suppliers will be under even more intense pressure from their investors than usual. There will most certainly be cost cutting at these firms, and if the concerns over the virus persist, it will lead to layoffs.
  • The smallest suppliers could face an existential risk. Many independent contractors and small firms are dependent on one or two clients for the bulk of their revenue. If those clients are in highly affected sectors, these small suppliers will be at risk of going out of business.
  • Smallish to mid-sized suppliers may emerge stronger. Clients are going to be under cost pressures due to a receding economy and smaller research suppliers tend to be less expensive. Smaller research firms did well post 9/11 and during the recession of 2008-09 because clients moved work from higher priced larger firms to them. Smaller research firms would be wise to build tight relationships so that when the storm over the virus abates, they will have won their clients trust for future projects.
  • New small firms will emerge as larger firms cut staff and create refugees who will launch new companies.

Those are all items that might pertain to any sort of sudden business downturn. There are also some things that we think will happen that are specific to the COVID-19 situation:

  • Market research conferences will never be the same. Conferences are going to have difficulty drawing speakers and attendees. Down the line, conferences will be smaller and more targeted and there will be more virtual conferences and training sessions scheduled. At a minimum, companies will send fewer people to research conferences.
  • This will greatly affect MR trade associations as these conferences are important revenue sources for them. They will rethink their missions and revenue models, and will become less dependent on their signature events. The associations will have more frequent, smaller, more targeted online events. The days of the large, comprehensive research conference may be over.
  • Business travel will not return to its previous level. There will be fewer in-person meetings between clients and suppliers and those that are held will have fewer participants. Video conferencing will become an even more important way to reach clients.
  • Clients and suppliers will allow much more “work from home.” It may become the norm that employees are only expected to be in the office for key meetings. The situation with COVID-19 will give companies who don’t have a lot of experience allowing employees to work from home the opportunity to see the value in it. When the virus is under control, they will embrace telecommuting. We will see this crisis kick-start an already existing movement towards allowing more employees to work from home. The amount of office space needed will shrink.
  • Research companies will review and revise their sick-leave policies and there will be pressure on them to make them more generous.
  • Companies that did the right thing during the crisis will be rewarded with employee loyalty. Employees will become more attached and appreciative of suppliers that showed flexibility, did what they could to maintain payroll, and expressed genuine concerns for their employees.

Probably the biggest change we will see in market research projects is to qualitative research.

  • While there will always be great value in traditional, in-person focus groups , the situation around COVID-19 is going to cause online qualitative to become the standard approach. We are at a time where the technologies available for online qualitative are well-developed, yet clients and suppliers have clung to traditional methods. To date, the technology has been ahead of the demand. Companies will be forced by travel restrictions to embrace online methods and this will be at the expense of traditional groups. This is an excellent time to be in the online qualitative technology business. It is not such a great time to be in the focus group facility management business.
  • Independent moderators, who work exclusively with traditional groups, are going to be in trouble and not just in the short term. Many of these individuals will retire or look for work elsewhere or leave research. Others will necessarily adapt to online methods. Of course, there will continue to be independent moderators but we are predicting the demand for in-person groups will be permanently affected, and this portion of the industry will significantly shrink.
  • There is a risk that by not commissioning as much in-person qualitative, marketers may become further removed from direct human interaction with their customer base. This is a very real concern. We wouldn’t be in market research if we didn’t have an affinity for data and algorithms, but qualitative research is what keeps all of our efforts grounded. I’d caution clients to think carefully before removing all in-person interaction from your research plans.

What will happen to quantitative research? In the short-run, most studies will continue. Respondents are home, have free time, and thus far have shown they are willing to take part in studies. Some projects, typically in highly affected industries like travel and entertainment, are being postponed or canceled. All current data sets need to be viewed with a careful eye as the tumult around the virus can affect results. For instance, we conduct a lot of research with young respondents, and we now know for sure that their parents are likely nearby when they are taking our surveys, and that can influence our findings for some subjects.

Particular care needs to be taken in ongoing tracking studies. It makes sense for many trackers to add questions in to see how the situation has affected the brand in question.

But, in the longer term there will be too much change in quantitative research methods that result directly from this situation. If anything, there will be a greater need to understand consumers.

Tough times for sure. It has been heartening to see how our industry has reacted. Research panel and technology providers have reached out to help keep projects afloat. We’ve had subcontractors tell us we can delay payments if we need to. Calls with clients have become more “human” as we hear their kids and pets in the background and see the stresses they are facing. Respondents have continued to fill out our surveys.

There is a lot of uncertainty right now. At its core, market research is a way to reduce uncertainty for decision makers by making the future more predictable, so we are needed now more than ever. Research will adapt as it always does, and I believe in the long-run it may become even more valued as a result of this crisis.

The myth of the random sample

Sampling is at the heart of market research. We ask a few people questions and then assume everyone else would have answered the same way.

Sampling works in all types of contexts. Your doctor doesn’t need to test all of your blood to determine your cholesterol level – a few ounces will do. Chefs taste a spoonful of their creations and then assume the rest of the pot will taste the same. And, we can predict an election by interviewing a fairly small number of people.

The mathematical procedures that are applied to samples that enable us to project to a broader population all assume that we have a random sample. Or, as I tell research analysts: everything they taught you in statistics assumes you have a random sample. T-tests, hypotheses tests, regressions, etc. all have a random sample as a requirement.

Here is the problem: We almost never have a random sample in market research studies. I say “almost” because I suppose it is possible to do, but over 30 years and 3,500 projects I don’t think I have been involved in even one project that can honestly claim a random sample. A random sample is sort of a Holy Grail of market research.

A random sample might be possible if you have a captive audience. You can random sample some the passengers on a flight or a few students in a classroom or prisoners in a detention facility. As long as you are not trying to project beyond that flight or that classroom or that jail, the math behind random sampling will apply.

Here is the bigger problem: Most researchers don’t recognize this, disclose this, or think through how to deal with it. Even worse, many purport that their samples are indeed random, when they are not.

For a bit of research history, once the market research industry really got going the telephone random digit dial (RDD) sample became standard. Telephone researchers could randomly call land line phones. When land line telephone penetration and response rates were both high, this provided excellent data. However, RDD still wasn’t providing a true random, or probability sample. Some households had more than one phone line (and few researchers corrected for this), many people lived in group situations (colleges, medical facilities) where they couldn’t be reached, some did not have a land line, and even at its peak, telephone response rates were only about 70%. Not bad. But, also, not random.

Once the Internet came of age, researchers were presented with new sampling opportunities and challenges. Telephone response rates plummeted (to 5-10%) making telephone research prohibitively expensive and of poor quality. Online, there was no national directory of email addresses or cell phone numbers and there were legal prohibitions against spamming, so researchers had to find new ways to contact people for surveys.

Initially, and this is still a dominant method today, research firms created opt-in panels of respondents. Potential research participants were asked to join a panel, filled out an extensive demographic survey, and were paid small incentives to take part in projects. These panels suffer from three response issues: 1) not everyone is online or online at the same frequency, 2) not everyone who is online wants to be in a panel, and 3) not everyone in the panel will take part in a study. The result is a convenience sample. Good researchers figured out sophisticated ways to handle the sampling challenges that result from panel-based samples, and they work well for most studies. But, in no way are they a random sample.

River sampling is a term often used to describe respondents who are “intercepted” on the Internet and asked to fill out a survey. Potential respondents are invited via online ads and offers placed on a range of websites. If interested, they are typically pre-screened and sent along to the online questionnaire.

Because so much is known about what people are doing online these days, sampling firms have some excellent science behind how they obtain respondents efficiently with river sampling. It can work well, but response rates are low and the nature of the online world is changing fast, so it is hard to get a consistent river sample over time. Nobody being honest would ever use the term “random sampling” when describing river samples.

Panel-based samples and river samples represent how the lion’s share of primary market research is being conducted today. They are fast and inexpensive and when conducted intelligently can approximate the findings of a random sample. They are far from perfect, but I like that the companies providing them don’t promote them as being random samples. They involve some biases and we deal with these biases as best we can methodologically. But, too often we forget that they violate a key assumption that the statistical tests we run require: that the sample is random. For most studies, they are truly “close enough,” but the problem is we usually fail to state the obvious – that we are using statistical tests that are technically not appropriate for the data sets we have gathered.

Which brings us to a newer, shiny object in the research sampling world: ABS samples. ABS (addressed-based samples) are purer from a methodological standpoint. While ABS samples have been around for quite some time, they are just now being used extensively in market research.

ABS samples are based on US Postal Service lists. Because USPS has a list of all US households, this list is an excellent sampling frame. (The Census Bureau also has an excellent list, but it is not available for researchers to use.) The USPS list is the starting point for ABS samples.

Research firms will take the USPS list and recruit respondents from it, either to be in a panel or to take part in an individual study. This recruitment can be done by mail, phone, or even online. They often append publicly-known information onto the list.

As you might expect, an ABS approach suffers from some of the same issues as other approaches. Cooperation rates are low and incentives (sometimes large) are necessary. Most surveys are conducted online, and not everyone in the USPS list is online or has the same level of online access. There are some groups (undocumented immigrants, homeless) that may not be in the USPS list at all. Some (RVers, college students, frequent travelers) are hard to reach. There is evidence that ABS approaches do not cover rural areas as well as urban areas. Some households use post office boxes and not residential addresses for their mail. Some use more than one address. So, although ABS lists cover about 97% of US households, the 3% that they do not cover are not randomly distributed.

The good news is, if done correctly, the biases that result from an ABS sample are more “correctable” than those from other types of samples because they are measurable.

A recent Pew study indicates that survey bias and the number of bogus respondents is a bit smaller for ABS samples than opt-in panel samples.

But ABS samples are not random samples either. I have seen articles that suggest that of all those approached to take part in a study based on an ABS sample, less than 10% end up in the survey data set.

The problem is not necessarily with ABS samples, as most researchers would concur that they are the best option we have and come the closest to a random sample. The problem is that many firms that are providing ABS samples are selling them as “random samples” and that is disingenuous at best. Just because the sampling frame used to recruit a survey panel can claim to be “random” does not imply that the respondents you end up in a research database constitute a random sample.

Does this matter? In many ways, it likely does not. There are biases and errors in all market research surveys. These biases and errors vary not just by how the study was sampled, but also by the topic of the question, its tone, the length of the survey, etc. Many times, survey errors are not the same throughout an individual survey. Biases in surveys tend to be “unknown knowns” – we know they are there, but aren’t sure what they are.

There are many potential sources of errors in survey research. I am always reminded of a quote from Humphrey Taylor, the past Chairman of the Harris Poll who said “On almost every occasion when we release a new survey, someone in the media will ask, “What is the margin of error for this survey?” There is only one honest and accurate answer to this question — which I sometimes use to the great confusion of my audience — and that is, “The possible margin of error is infinite.”  A few years ago, I wrote a post on biases and errors in research, and I was able to quickly name 15 of them before I even had to do an Internet search to learn more about them.

The reality is, the improvement in bias that is achieved by an ABS sample over a panel-based sample is small and likely inconsequential when considered next to the other sources of error that can creep into a research project. Because of this, and the fact that ABS sampling is really expensive, we tend to only recommend ABS panels in two cases: 1) if the study will result in academic publication, as academics are more accepting of data that comes from and ABS approach, and 2) if we are working in a small geography, where panel-based samples are not feasible.

Again, ABS samples are likely the best samples we have at this moment. But firms that provide them are often inappropriately portraying them as yielding random samples. For most projects, the small improvements in bias they provide is not worth the considerable increased budget and increased study time frame, which is why, for the moment, ABS samples are currently used in a small proportion of research studies. I consider ABS to be “state of the art” with the emphasis on “art” as sampling is often less of a science than people think.


Visit the Crux Research Website www.cruxresearch.com

Enter your email address to follow this blog and receive notifications of new posts by email.