Archive for the 'Methodology' Category

Should we get rid of statistical significance?

There has been recent debate among academics and statisticians surrounding the concept of statistical significance. Some high-profile medical studies have just narrowly missed meeting the traditional statistical significance cutoff of 0.05. This has resulted in potentially life changing drugs not being approved by regulators or pursued for further development by pharma companies. These cases have led to a much-needed review and re-education as to what statistical significance means and how it should be applied.

In a 2014 blog post (Is This Study Significant?) we discussed common misunderstandings market researchers have regarding statistical significance. The recent debate suggests this misunderstanding isn’t limited to market researchers – it appears that academics and regulators have the same difficulty.

Statistical significance is a simple concept. However, it seems that the human brain just isn’t wired well to understand probability and that lies at the root of the problem.

A measure is typically classified as statistically significant if its p-value is 0.05 or less. This means that there is a less than 5% probability that the result came from chance or random fluctuation. Two measures are deemed to be statistically different if there is a 19 out of 20 chance or greater that they are.

There are real problems with this approach. Foremost, it is unclear how this 5% probability cutoff was chosen. Somewhere along the line it became a standard among academics. This standard could have just as easily been 4% or 6% or some other number. This cutoff was chosen subjectively.

What are the chances that this 5% cutoff is optimal for all studies, regardless of the situation?

Regulators should look beyond statistical significance when they are reviewing a new medication. Let’s say a study was only significant at 6%, not quite meeting the 5% standard. That shouldn’t automatically disqualify a promising medication from consideration. Instead, regulators should look at the situation more holistically. What will the drug do? What are its side effects? How much pain does it alleviate? What is the risk of making mistakes in approval: in approving a drug that doesn’t work or in failing to approve a drug that does work? We could argue that the level of significance required in the study should depend on the answers to these questions and shouldn’t be the same in all cases.

The same is true in market research. Suppose you are researching a new product and the study is only significant at 10% and not the 5% that is standard. Whether you should greenlight the product for development depends on considerations beyond statistical significance. What is the market potential of the product? What is the cost of its development? What is the risk of failing to greenlight a winning idea or greenlighting a bad idea? Currently, too many product managers rely too much on a research project to give them answers when the study is just one of many inputs into these decisions.

There is another reason to rethink the concept of statistical significance in market research projects. Statistical significance assumes a random or a probability sample. We can’t stress this enough – there hasn’t been a market research study conducted in at least 20 years that can credibly claim to have used a true probability sample of respondents. Some (most notably ABS samples) make a valiant attempt to do so but they still violate the very basis for statistical significance.

Given that, why do research suppliers (Crux Research included) continue to do statistical testing on projects? Well, one reason is clients have come to expect it. A more important reason is that statistical significance holds some meaning. On almost every study we need to draw a line and say that two data poworints are “different enough” to point out to clients and to draw conclusions from. Statistical significance is a useful tool for this. It just should no longer be viewed as a tool where we can say precise things like “these two data points have a 95% chance of actually being different”.

We’d rather use a probability approach and report to clients the chance that two data points would be different if we had been lucky enough to use a random sample. That is a much more useful way to look at data, but it probably won’t be used much until colleges start teaching it and a new generation of researchers emerges.

The current debate over the usefulness of statistical significance is a healthy one to have. Hopefully, it will cause researchers of all types to think deeper about how precise a study needs to be and we’ll move away from the current one-size-fits-all thinking that has been pervasive for decades.

Jeff Bezos is right about market research

In an annual shareholder letter, Amazon’s Jeff Bezos recently stated that market research isn’t helpful. That created some backlash among researchers, who reacted defensively to the comment.

For context, below is the text of Bezos’ comment:

No customer was asking for Echo. This was definitely us wandering. Market research doesn’t help. If you had gone to a customer in 2013 and said “Would you like a black, always-on cylinder in your kitchen about the size of a Pringles can that you can talk to and ask questions, that also turns on your lights and plays music?” I guarantee you they’d have looked at you strangely and said “No, thank you.”

This comment is reflective of someone who understands the role market research can play for new products as well as its limitations.

We have been saying for years that market research does a poor job of predicting the success of truly breakthrough products. What was the demand for television sets in the 1920’s and 1930’s before there was even content to broadcast or a way to broadcast it? Just a decade ago, did consumers know they wanted a smartphone they would carry around with them all day and constantly monitor? Henry Ford once said that if he had asked customers what they wanted they would have wanted faster horses and not cars.

In 2014, we wrote a post (Writing a Good Questionnaire is Just Like Brian Surgery) that touched on this issue. In short, consumer research works best when the consumer has a clear frame-of-reference from which to draw. New product studies on line extensions or easily understandable and relatable new ideas tend to be accurate. When the new product idea is harder to understand or is outside the consumer’s frame-of-reference research isn’t as predictive.

Research can sometimes provide the necessary frame-of-reference. We put a lot of effort to be sure that concept descriptions are understandable. We often go beyond words to do this and produce short videos instead of traditional concept statements. But even then, if the new product being tested is truly revolutionary the research will probably predict demand inaccurately. The good news is few new product ideas are actually breakthroughs – they are usually refinements on existing ideas.

Failure to provide a frame-of-reference or realize that one doesn’t exist leads to costly research errors. Because this error is not quantifiable (like a sample error) it gets little attention.

The mistake people are making when reacting to Bezos’ comment is they are viewing it as an indictment of market research in general. It is not. Research still works quite well for most new product forecasting studies. For new products, companies are often investing millions or tens of millions in development, production, and marketing. It usually makes sense to invest in market research to be confident these investments will pay off and to optimize the product.

It is just important to recognize that there are cases where respondents don’t have a good frame-of-reference and the research won’t accurately predict demand. Truly innovative ideas are where this is most likely to happen.

I’ve learned recently that this anti-research mentality pervades the companies in Silicon Valley. Rather than use a traditional marketing approach of identifying a need and then developing a product to fulfill the need, tech firms often concern themselves first with the technology. They develop a technology and then look for a market for it. This is a risky strategy and likely fails more than it succeeds, but the successes, like the Amazon Echo, can be massive.

I own an Amazon Echo. I bought it shortly after it was launched having little idea what it was or what it could do. Even now I am still not quite sure what it is capable of doing. It probably has a lot of potential that I can’t even conceive of. I think it is still the type of product that might not be improved much by market research, even today, when it has been on the market for years.

Will adding a citizenship question to the Census harm the Market Research Industry?

The US Supreme Court appears likely to allow the Department of Commerce to reinstate a citizenship question on the 2020 Census. This is largely viewed as a political controversy at the moment. The inclusion of a citizenship question has proven to dampen response rates among non-citizens, who tend to be people of color. The result will be gains in representation for Republicans at the expense of Democrats (political district lines are redrawn every 10 years as a result of the Census). Federal funding will likely decrease for states with large immigrant populations.

It should be noted that the Census bureau itself has come out against this change, arguing that it will result in an undercount of about 6.5 million people. Yet, the administration has pressed forward and has not committed funds needed by the Census Bureau to fully research the implications. The concern isn’t just about non-response from non-citizens. In tests done by the Census Bureau, non-citizens are also more likely to inaccurately respond to this question than citizens, meaning the resulting data will be inaccurate.

Clearly this is a hot-button political issue. However, there is not much talk of how this change may affect research. Census data are used to calibrate most research studies in the US, including academic research, social surveys, and consumer market research. Changes to the Census may have profound effects on data quality.

The Census serves as a hidden backbone for most research studies whether researchers or clients realize it or not. Census information helps us make our data representative. In a business climate that is becoming more and more data-driven the implications of an inaccurate Census are potentially dire.

We should be primarily concerned that the Census is accurate regardless of the political implications. Adding questions that temper response will not help accuracy. Errors in the Census have a tendency to become magnified in research. For example, in new product research it is common to project study data from about a thousand respondents to a universe of millions of potential consumers. Even a small error in the Census numbers can lead businesses to make erroneous investments. These errors create inefficiencies that reverberate throughout the economy. Political concerns aside, US businesses undoubtably suffer from a flawed Census. Marketing becomes less efficient.

All is not lost though. We can make a strong case that there are better, less costly ways to conduct the Census. Methodologists have long suggested that a sampling approach would be more accurate than the current attempt at enumeration. This may never happen for the decennial Census because the Census methodology is encoded in the US Constitution and it might take an amendment to change it.

So, what will happen if this change is made? I suspect that market research firms will switch to using data that come from the Census’ survey programs, such as the American Community Survey (ACS). Researchers will rely less on the actual decennial census. In fact, many research firms already use the ACS rather than the decennial census (and the ACS currently contains the citizenship question).

The Census bureau will find ways to correct for resulting error, and to be honest, this may not be too difficult from a methodological standpoint. Business will adjust because there will be economic benefits to learning how to deal with a flawed Census, but in the end, this change will take some time for the research industry to address. Figuring things like this out is what good researchers do. While it is unfortunate that this change looks likely to be made, its implications are likely more consequential politically than it will be to the research field.

Long Live the Focus Group!

Market research has changed over the past two decades. Telephone research has faded away, mail studies are rarely considered, and younger researchers have likely never conducted a central location test in a mall. However, there is an old-school type of research that has largely survived this upheaval:  the traditional, in-person focus group.

There has been extensive technological progress in qualitative research. We can now conduct groups entirely online, in real-time, with participants around the globe. We can conduct bulletin board style online groups that take place over days. Respondents can respond via text or live video, can upload assignments we give them, and can take part in their own homes or workplaces. We can intercept them when they enter a store and gather insights “in the moment.” We even use technology to help make sense of the results, as text analytics has come a long way and is starting to prove its use in market research.

These new, online qualitative approaches are very useful. They save on travel costs, can be done quickly, and are often less expensive than traditional focus groups. But we have found that they are not a substitute for traditional focus groups, at least not in the way that online surveys have substituted for telephone surveys. Instead, online qualitative techniques are new tools that can do new things, but traditional focus groups are still the preferred method for many projects.

There is just no real substitute for the traditional focus group that allows clients to see actual customers interact around their product or issue. In some ways, as our world has become more digital traditional focus groups provide a rare opportunity to see and hear from customers. They are often the closest clients get to actually seeing their customers in a live setting.

I’ve attended hundreds of focus groups. I used to think that the key to a successful focus group was the skill of the moderator followed by a cleverly designed question guide. Clients spend a lot of time on the question guide. But they spend very little time on something that is critical to every group’s success: the proper screening of participants.

Seating the right participants is every bit as important as constructing a good question guide. Yet, screening is given passing attention by researchers and clients. Typically, once we decide to conduct groups a screener is turned around within a day because we need to get moving on the recruitment. In contrast, a discussion guide is usually developed over a full week or two.

Developing an outstanding screener starts by having a clear sense of objectives. What decisions are being made as a result of the project? Who is making them? What is already known? How will the decision path differ based on what we find? I am always surprised that in probably half of our qualitative projects our clients don’t have answers to these questions.

Next, it is important to remind clients that focus groups are qualitative research and we shouldn’t be attempting to gather a “representative” sample. Focus groups happen with a limited number of participants in a handful of cities and we shouldn’t be trying to project findings to a larger audience. If that is needed, a follow-up quantitative phase is required. Instead, in groups we are trying to delve deeply into motivations, explore ideas, and develop with new hypotheses we can test later.

It is a common mistake to try to involve enough participants to make findings “valid.” This is important, as we are looking for thoughtful participants and not necessarily “typical” customers. We want folks that will expand our knowledge of a subject and of customers will help us explore deeply into topics and develop new lines of inquiry we haven’t considered.

“Representative” participants can be quiet and reserved and not necessarily useful to this phase of research. For this reason, we always use articulation screening questions which raise the odds that we will get a talkative participant who enjoys sharing his/her opinions.

An important part of the screening process is determining how to segment the groups. It is almost never a good idea to hold all of your sessions with the same audience. We tend to segment on age, potentially gender, and often by the participants’ experience level with the product or issue. Contrasting findings from these groups is often where the key qualitative insights lie.

It is also necessary to over-recruit. Most researchers overrecruit to protect against participants who fail to show up to the sessions. We do it for another reason. We like to have a couple of extra participants in the waiting area. Before the groups start, the moderator spends some time with them. This accomplishes two things. First, the groups are off and running the moment participants enter the focus group room because a rapport with the moderator has been established. Second, spending a few minutes with participants before groups begin allows the moderator to determine in advance which participants are going to be quiet or difficult, and allows us to pay them the incentive and send them home.

Clients tend to insist on group sizes that are too large. I have viewed groups with as many as 12 respondents. Even in a two-hour session, the average participant will be talking for just 10 minutes in this case and that is if there are no silences or the moderator doesn’t talk! In reality, with 12 participants you will get maybe five minutes out of each one. How is that useful?

Group dynamics are different in smaller groups. We like to target having about six participants. This group size is small enough that all must participate and engage, but large enough to get a diversity of views.  We also prefer to have groups run for 90 minutes or less.

We like to schedule some downtime in between groups. The moderator needs this to recharge (and eat!), but this also gives time for a short debrief and to adjust the discussion guide on the fly. I have observed groups where the moderator is literally doing back-to-back sessions for six hours and it isn’t productive. Similarly, it is ideal to have a rest day in between cities to regroup to provide an opportunity to develop new questions. (Although, this is rarely done in practice.)

Clients also need to learn to leave the moderator alone for at least 30 minutes before the first group begins. Moderating is stressful, even for moderators who have led thousands of groups. They need time to review the guide and converse with the participants. Too many times, clients are peppering the moderator with last second changes to the guide and in general are stressing the moderator right before the first session. These discussions need to be held before focus group day.

We’d also caution against conducting too many groups. I remember working on a proposal many years ago when our qualitative director was suggesting we conduct 24 focus groups. She was genuinely angry at me when I asked her “what are we going to learn in that 24th group that we didn’t learn in the first 23?”.

In all candor, in my experience you learn about 80% of what you will learn in the first evening of groups. It is useful to conduct another evening or two to confirm what you have heard. But it is uncommon for a new insight arises after the first few groups. It is a rare project that needs more than about two cities’ worth of groups.

It is also critical to have the right people from the clients attending the sessions. With the right people present discussions behind the mirror become insightful and can be the most important part of the project. Too often, clients send just one or two people from the research team and the internal decision makers stay home. I have attended groups where the client hasn’t shown up at all and it is just the research supplier who is there. If the session isn’t important enough to send decision makers to attend, it probably isn’t important enough to be doing in the first place.

I have mixed feelings about live streaming sessions. This can be really expensive and watching the groups at home is not the same as being behind the mirror with your colleagues. Live streaming is definitely better than not watching them at all. But I would say about half the time our clients pay for live streaming nobody actually logs in to watch them.

Focus groups are often a lead-in to a quantitative study. We typically enter into the groups with an outline of the quantitative questionnaire at the ready. We listen purposefully at the sessions to determine how we need to refine our questionnaire. This is more effective than waiting for the qualitative to be over before starting the quantitative design. We can usually have the quant questionnaire ready for review before the report for the groups is available because we take this approach.

Finally, it is critical to debrief at the end of each evening. This is often skipped. Everyone is tired, has been sitting in the dark for hours, and have to get back to a hotel and get up early for a flight. But, a quick discussion to agree on the key takeaways while they are fresh in mind is very helpful. We try to get clients to agree to these debriefings before the groups are held.

Traditional groups provide more amazing moments and unexpected insights than any other research method. I think this may be why, despite all the new options for qualitative, clients are conducting just as many focus groups as ever.

Is segmentation just discrimination with an acceptable name?

A short time ago we posted a basic explanation of the Cambridge Analytica/Facebook scandal (which you can read here). In it, we stated that market segmentation and stereotyping are essentially the same thing. This presents an ethical quandary for marketers as almost every marketing organization makes heavy use of market segmentation.

To review, marketers place customers into segments so that they can better understand and serve them. Segmentation is at the essence of marketing. Segments can be created along any measurable dimension, but since almost all segments have a demographic component we will focus on that for this post.

It can be argued that segmentation and stereotyping are the same thing. Stereotyping is attaching perceived group characteristic to an individual. For instance, if you are older I might assume your political views lean conservative, since it is known that political views tend to be more conservative in older Americans that they are in general among younger Americans. If you are female I might assume you are more likely to be the primary shopper for your household, since females in total do more of the family shopping than males. If you are African-American, I might assume you have a higher likelihood than others to listen to rap music, since that genre indexes high among African-Americans.

These are all stereotypes. These examples can be shown to true of a larger group, but that doesn’t necessarily imply that they apply to all the individuals in the group. There are plenty of liberal older Americans, females who don’t shop at all, and African-Americans who can’t stand rap music.

Segmenting consumers (which is applying stereotypes) isn’t inherently a bad thing. It leads to customized products and better customer experiences. The potential problem isn’t with stereotyping, it is when doing so moves to a realm of being discriminatory that we have to be careful. As marketers we tread a fine line. Stereotyping oversimplifies the complexity of consumers by forming an easy to understand story. This is useful in some contexts and discriminatory in others.

Some examples are helpful. It can be shown that African-Americans have a lower life expectancy than Whites. A life insurance company could use this information to charge African-Americans higher premiums than Whites. (Indeed, many insurance companies used to do this until various court cases prevented them from doing so.) This is a segmentation practice that many would say crosses a line to become discriminatory.

In a similar vein, car insurance companies routinely charge higher risk groups (for example younger drivers and males) higher rates than others. That practice has held up as not being discriminatory from a legal standpoint, largely because the discrimination is not against a traditionally disaffected group.

At Crux, we work with college marketers to help them make better admissions offer decisions. Many colleges will document the characteristics of their admitted students who thrive and graduate in good standing. The goal is to profile these students and then look back at how they profiled as applicants. The resulting model can be used to make future admissions decisions. Prospective student segments are established that have high probabilities of success at the institution because they look like students known to be successful, and this knowledge is used to make informed admissions offer decisions.

However, this is a case where a segmentation can cross a line and become discriminatory. Suppose that the students who succeed at the institution tend to be rich, white, female, and from high performing high schools. By benchmarking future admissions offers against them, an algorithmic bias is created. Fewer minorities, males, and students from urban districts will be extended admissions offers What turns out to be a good model from a business standpoint ends up perpetuating a bias., and places certain demographics of students at a further disadvantage.

There is a burgeoning field in research known as “predictive analytics.” It allows data jockeys to use past data and artificial intelligence to make predictions on how consumers will react. It is currently mostly being used in media buying. Our view is it helps in media efficiency, but only if the future world can be counted on to behave like the past. Over-reliance on predictive analytics will result in marketers missing truly breakthrough trends. We don’t have to look further than the 2016 election to see how it can fail; many pollsters were basing their modeling on how voters had performed in the past and in the process missed a fundamental shift in voter behavior and made some very poor predictions.

That is perhaps an extreme case, but shows that segmentations can have unintended consequences. This can happen in consumer product marketing as well. Targeted advertising can become formulaic. Brands can decline distribution in certain outlets. Ultimately, the business can suffer and miss out on new trends.

Academics (most notably Kahneman and Tversky) have established that people naturally apply heuristics to decision making. These are “rules of thumb” that are often useful because they allow us to make decisions quickly. However, these academics have also demonstrated how the use of heuristics often result in sub-optimal and biased decision making.

This thinking applies to segmentation. Segmentation allows us to make marketing decisions quickly because we assume that individuals take on the characteristics of a larger group. But, it ignores the individual variability within the group, and often that is where the true marketing insight lies.

We see this all the time in the generational work we do. Yes, Millennials as a group tend to be a bit sheltered, yet confident and team-oriented. But this does not mean all of them fit the stereotype. In fact, odds are high that if you profile an individual from the Millennial generation, he/she will only exhibit a few of the characteristics commonly attributed to the generation. Taking the stereotype too literally can lead to poor decisions.

This is not to say that marketers shouldn’t segment their customers. This is a widespread practice that clearly leads to business results. But, they should do so considering the errors and biases applying segments can create, and think hard about whether this can unintentionally discriminate and, ultimately, harm the business in the long term.

Has market research become Big Brother?

Technological progress has disrupted market research. Data are available faster and cheaper than ever before. Many traditional research functions have been automated out of existence or have changed significantly because of technology. Projects take half the time to complete that they did just a decade ago. Decision making has moved from an art to a science. Yet, as with most technological disruptions, there are just as many potential pitfalls as efficiencies to be wary of as technology changes market research.

“Passive” data collection is one of these potential pitfalls. It is used by marketers in good ways: the use of passive data helps understand consumers better, target meaningful products and services, and create value for both the consumer and the marketer. However, much of what is happening with passive data collection is done without the full knowledge of the consumer and this process has the potential of being manipulative. The likelihood of backlash towards the research industry is high.

The use of passive data in marketing and research is new and many researchers may not know what is happening so let us explain. A common way to obtain survey research respondents is to tap into large, opt-in online panels that have been developed by a handful of companies. These panels are often augmented with social (river) channels whereby respondents are intercepted while taking part in various online activities. A recruitment email or text is delivered, respondents take a survey, and data are analyzed. Respondents provide information actively and with full consent.

There have been recent mergers which have resulted in fewer but larger and more robust online research panels available. This has made it feasible for some panel companies to gain the scale necessary to augment this active approach with passive data.

It is possible to append information from all sorts of sources to an online panel database. For instance, voter registration files are commonly appended. If you are in one of these research panels, clients likely know if you are registered to vote, if you actually voted, and your political party association. They will have made a prediction of how strong a liberal or conservative you likely are. They may have even run models to predict which issues you care most about. You are likely linked into a PRIZM cluster that associates you with characteristics of the neighborhood where you reside, which in turn can score your potential to be interested in all sorts of product categories. This is all in your file.

These panels also have the potential to link to other publicly-available databases such as car registration files, arrest records, real estate transactions, etc. If you are in these panels, whether you have recently bought a house, how much you paid for it, if you have been convicted of a crime, may all be in your “secret file.”

But, it doesn’t stop there. These panels are now cross-referenced to other consumer databases. There are databases that gather the breadcrumbs you leave behind in your digital life: sites you are visiting, ads you have been served, and even social media posts you have made. There is a tapestry of information available that is far more detailed than most consumers realize. From the research panel company’s perspective, it is just a matter of linking that information to their panel.

This opens up exciting research possibilities. We can now conduct a study among people who are verified to have been served by a specific client’s digital advertising. We can refine our respondent base further by those who are known to have clicked on the ad. As you can imagine, this can take ad effectiveness research to an entirely different level. It is especially interesting to clients because it can help optimize media spending which is by far the largest budget item for most marketing departments.

But, therein lies the ethical problem. Respondents, regardless of what privacy policies they may have agreed to, are unlikely to know that their passive web behavior is being linked into their survey responses. This alone should ring alarm bells for an industry suffering from low response rates and poor data quality. Respondents are bound to push back when they realize there is a secret file panel companies are holding on them.

Panel companies are straying from research into marketing. They are starting to encourage clients to use the survey results to better target individual respondents in direct marketing. This process can close a loop with a media plan. So, say on a survey you report that you prefer a certain brand of a product. That can now get back to you and you’ll start seeing ads for that product, likely without your knowledge that this is happening because you took part in a survey.

To go even further, this can affect advertising people not involved in the survey may see. If you prefer a certain brand and I profile a lot like you, as a result of your participation in a survey I may end up seeing specific ads. Even if I don’t know you or have any connection to you.

In some ways, this reeks of the Cambridge Analytica scandal (which we explain in a blog post here). We’ll be surprised if this practice doesn’t eventually create a controversy in the survey research industry. This sort of sales targeting resulting from survey participation will result in lower response rates and a further erosion of confidence in the market research field. However, it is also clear that these approaches are inevitable and will be used more and more as panel companies and clients gain experience with them.

It is the blurring of the line between marketing and market research that has many old-time researchers nervous. There is a longstanding ethical tenet in the industry that participation in research project should in no way result in the respondent being sold or marketed to. The term for this is SUGGING (Selling Under the Guise of research) and all research industry trade groups have a prohibition against SUGGING embedded in their codes of ethics. It appears that some research firms are ignoring this. But, this concept has always been central to the market research field: we have traditionally assured respondents that they can be honest on our surveys because we will in no way market to them directly because of their answers.

In the novel 1984 George Orwell describes a world where the government places its entire civilization under video surveillance. For most of the time since its publication, this has appeared as a frightening, far-fetched cautionary tale. Recent history has suggested this world may be upon us. The NSA scandal (precipitated by Edward Snowden) showed how much of our passive information is being shared with the government without our knowledge. Rather than wait for the government to surveil the population, we’ve turned the cameras on ourselves. Marketers can do things I don’t feel people realize and research respondents are unknowingly enabling this. The contrails you leave as you simply navigate your life online can be used to follow you and the line between research and marketing is fading, and this will eventually be to the detriment of our field.

Market research isn’t about storytelling, it is about predicting the future

We recently had a situation that made me question the credibility of market research. We had fielded a study for a long-term client and were excited to view the initial version of the tabs. As we looked at results by age groupings we found them to be surprising. But this was also exciting because we were able to weave a compelling narrative around why the age results seemed counter-intuitive.

Then our programmer called to say a mistake had been made in the tabs and the banner points by age had been mistakenly reversed.

So, we went back to the drawing board ad constructed another, equally compelling story, as to why the data were behaving as they were.

This made me question the value of research. Good researchers can review seemingly disparate data points from a study and generate a persuasive story as to why they are as they are. Our entire business is based on this skill – in the end clients pay us to use data to provide insight into their marketing issues. Everything else we do is a means to this end.

Our experience with the flipped age banner points illustrates that stories can be created around any data. In fact, I’d bet that if you gave us a randomly-generated data set we could convince you as to its relevance to your marketing issues. I actually thought about doing this – taking the data we obtain by running random data through a questionnaire when testing it before fielding, handing it to an analyst, and seeing what happens. I’m convinced we could show you a random data set’s relevance to your business.

This issue is at the core of polling’s PR problem. We’ve all heard people say that you can make statistics say anything, therefore polls can’t be trusted. There are lies, damn lies, and statistics. I’ve argued against this for a long time because the pollsters and researchers I have known have universally been well-intentioned and objective and never try to draw a pre-determined conclusion from the data.

Of course, this does not mean that all of the stories we tell with data aren’t correct or enlightening. But, they all come from a perspective. Clients value external suppliers because of this perspective – we are third-party observers who aren’t wrapped up in the internal issues client’s face and we are often in a good position to view data with an objective mind. We’ve worked with hundreds of organizations and can bring these experiences bring that to bear on your study. Our perspective is valuable.

But, it is this perspective that creates an implicit bias in all we do. You will assess a data set from a different set of life experiences and background than I will. That is just human nature. Like all biases in research, our implicit bias may or not be relevant to a project. In most cases, I’d say it likely isn’t.

So, how can researchers reconcile this issue and sleep at night knowing their careers haven’t been a sham?

First and foremost, we need to stop saying that research is all about storytelling. It isn’t. The value of market research isn’t in the storytelling it is in the predictions of the future it makes. Clients aren’t paying us to tell them stories. They are paying us to predict the future and recommend actions that will enhance their business. Compelling storytelling is a means to this but is not our end goal. Data-based storytelling provides credibility to our predictions and gives confidence that they have a high probability of being correct.

In some sense, it isn’t the storytelling that matters, it is the quality of the prediction. I remember having a college professor lecturing on this. He would say that the quality of a model is judged solely by its predictive value. Its assumptions, arguments, and underpinnings really didn’t matter.

So, how do we deal with this issue … how do we ensure that the stories we tell with data are accurate and fuel confident predictions? Below are some ideas.

  1. Make predictions that can be validated at a later date. Provide a level of confidence or uncertainty around the prediction. Explain what could happen to prevent your prediction from coming true.
  2. Empathize with other perspectives when analyzing data. One of the best “tricks” I’ve ever seen is to re-write a research report as if you were writing it for your client’s top competitor. What conclusions would you draw for them? If it is an issue-based study, consider what you would conclude from the data if your client was on the opposite side of the issue.
  3. Peg all conclusions to specific data points in the study. Straying from the data is where your implicit bias may tend to take over. Being able to tie conclusions directly to data is dependent on solid questionnaire design.
  4. Have a second analyst review your work and play devil’s advocate. Show him/her the data without your analysis and see what stories and predictions he/she can develop independent of you. Have this same person review your story and conclusions and ask him/her to try to knock holes in them. The result is a strengthened argument.
  5. Slow down. It just isn’t possible to provide stories, conclusions, and predictions from research data that consider differing perspectives when you have just a couple of days to do it. This requires more negotiation upfront as to project timelines. The ever-decreasing timeframes for projects are making it difficult to have the time needed to objectively look at data.
  6. Realize that sometimes a story just isn’t there. Your perspective and knowledge of a client’s business should result in a story leaping out at you and telling itself. If this doesn’t happen, it could be because the study wasn’t designed well or perhaps there simply isn’t a story to be told. The world can be a more random place than we like to admit, and not everything you see in a data set is explainable. Don’t force it – developing a narrative that is reaching for explanations is inaccurate and a disservice to your client.

The Cambridge Analytica scandal points to marketing’s future

There has been a lot of press, almost universally bad, regarding Cambridge Analytica recently. Most of this discussion has centered on political issues (how their work may have benefitted the Trump campaign) and on data privacy issues (how this scandal has shined a light on the underpinnings of Facebook’s business model). One thing that hasn’t been discussed is the technical brilliance of this approach to combining segmentation, big data, and targeted communications to market effectively. In the midst of an incredibly negative PR story lurks the story of a controversial future of market research and marketing.

To provide a cursory and perhaps oversimplified recap of what happened, this all began with a psychographic survey which provided input into a segmentation. This is a common type of market research project. Pretty much every brand you can think of has done it. The design usually has a basis in psychology and the end goal is typically to create subgroups of consumers that provide a better customer understanding and ultimately help a client spend marketing resources more efficiently by targeting these subgroups.

Almost every marketer targets demographically – by easy to identify characteristics such as age, gender, race/ethnicity, and geography. Many also target psychographic ally – by personality characteristics and deeper psychological constructs. The general approach taken by Cambridge Analytics has been perfected over decades and is hardly new. I’d say I’ve been involved in about 100 projects that involve segmenting on a psychographic basis.

To give a concrete example, this type of approach is used by public health campaigns seeking to minimize drug and alcohol use. Studies will be done on a demographic basis that indicate things like drug use skews towards males more than females, towards particular age groups, and perhaps even certain regions of the country. But, it can also be shown that those most at risk of addiction also have certain personality types – they are risk takers, sensation seekers, extroverts, etc. Combined with demographic information, this can allow a public health marketer to target their marketing spend as well as help them craft messages that will resound with those most at risk.

Segmentation is essentially stereotyping with another name. It is associating perceived characteristics of a group with an individual. At its best, this approach can provide the consumer with relevant marketing and products customized to his/her needs. At its worst, it can ignore variation within a group and devalue the consumer as an individual. Segmentation can turn to prejudice and profiling fast and marketers can put too much faith in it.

Segmentation is imperfect. Just because you are a male, aged 15-17, and love to skateboard without a helmet and think jumping out of an airplane would be cool does not necessarily mean you are at risk to initiate drug use. But, our study might show that for every 100 people like you, 50 of them are at risk, and that is enough to merit spending prevention money towards reaching you. You might not be at risk for drug use, but we think you have a 50% chance of being so and this is much higher than the general risk in the population. This raises the efficiency of marketing spending.

What Cambridge Analytica did was analogous to this. The Facebook poll users completed provided data needed to establish segments. These segments were then used to predict your likelihood to care about an issue. Certain segments might be more associated with hot button issues in the election campaign, say gun rights, immigration, loss of American jobs, or health care. So, once you filled out the survey, combined with demographic data, it became possible to “score” you on these issues. You might not be a “gun nut” but your data can provide the researcher with the probability that you are, and if it is high enough you might get an inflammatory gun rights ad targeted to you.

Where this got controversial was, first and foremost, regardless of what Facebook’s privacy policy may say, most users had no clue that answering an innocuous quiz might enable them to be targeted in this way. Cambridge Analytica had more than the psychographic survey at their disposal – they also had demographics, user likes and preferred content, and social connections. They also had much of this information on the user’s Facebook friends as well. It is the depth of the information they gathered than has led to the crisis at Facebook.

People tend to associate most strongly with people who are like them. So, if I score you high on a “gun nut scale” chances are reasonably high that your close friends will have a high probability of being like you. So, with access to your friends, a marketer can greatly expand the targeted reach of the campaign.

It is hard to peel away from the controversies to see how this story really points to the future of marketing, and how research will point the way. Let me explain.

Most segmentations suffer from a fatal flaw: they segment with little ability to follow up by targeting. With a well-crafted survey we can almost always create segments help a marketer better understand his/her customers. But, often (and I would even say most of the time) it is next to impossible to target these segments. Back to the drug campaign example, since I know what shows various demographic groups watch, I can tell you to spend your ad dollars on males aged 16-17. But, how the heck do you then target further and find a way to reach the “risk taking” segment you really want? If you can’t target, segmentation is largely an academic exercise.

Traditionally you couldn’t target psychographic segments all that well. But, with what Google and Facebook now know about their users, you can. If we can profile enough of the Facebook teenage user base and have access to who their friends are, we can get incredibly efficient in our targeting.  Ad spend can get to those who have a much higher propensity for drug use and we can avoid wasting money on those who have low propensity.

It is a brilliant approach. But, like most things on the Internet, it can be a force for bad as well as good. If what Cambridge Analytica had done was for the benefit of an anti-drug campaign, I don’t think it would be nearly the story it has become. Once it went into a polarized political climate, it became news gold.

Even when an approach like this is applied to what most would call legitimate marketing, say for a consumer packaged good, it can get a bit creepy and feel manipulative. It is conceivable that via something one of my Facebook friends did, I can get profiled as a drinker of a specific brand of beer. Since Google also knows where my phone is, I can then be sent an ad or a coupon at the exact moment I walk by the beer case in my local grocery store. Or, my friends can be sent the same message. And I didn’t do anything to knowingly opt into being targeted like this.

There are ethical discussions that need to be had regarding whether this is good or bad, if it is a service to the consumer, or if it is too manipulative. But, this sort of targeting and meshing of research and marketing is not futuristic – all of the underpinning technology is there at the ready and it is only a matter of time until marketers really learn how to tap into it. It is a different world for sure and one that is coming fast.


Visit the Crux Research Website www.cruxresearch.com

Enter your email address to follow this blog and receive notifications of new posts by email.