Posts Tagged 'Crux Research'

Is getting a driver’s license still a rite of passage for teens?

In the 80’s and 90’s, before the Millennial generation hit their teen years in force, we would use “driver’s license status” as a key classification variable in studies. Rather than split focus groups by age or grade in school, we would often place teens who had their license in one group and those who did not have their license yet in another group. Regardless of the topic of the group. We found that teens with licenses were more independent of their parents and more capable of making decisions without parental input. Drivers license obtention was often better predictor of consumer behavior than age.

Young people experience many rites of passages in a short period of time. These are experiences that signify a change in their development. They ride the school bus for the first time, get their first smartphone, enter high school, go to the prom, leave home to go to college, vote for the first time, etc. As marketers, we have always looked at these inflection points as times when consumer behavior shifts. The obtaining of a driver’s license is traditionally seen as a watershed moment as it signifies a new level of independence.

However, this wisdom no longer holds. Millennials, particularly second wave Millennials, are not as focused on obtaining drivers licenses as their Boomer and Xer parents were. Where I grew up, we couldn’t wait until our 16th birthday so we could get our learner’s permit. My classmates and I usually took our road tests at the first opportunity. Failing the road test was a traumatic experience, as it caused us to remain in our parents’ control for a few more months.

This is no longer the case. In 1983, 46% of America’s 16-year-olds had a driver’s license. That is now less than 25% currently. I was very surprised to notice that my children and their friends seemed to be in no particular rush to get their licenses. Many times, it was the parents that pushed the kids to take their road test, as the parents were tiring of chaperoning the kids from place to place.

There are likely things that have caused this change:

  • Today’s parents are highly protective of children. Parents no longer push their children to be as independent as quickly.
  • There are societal pressures. In most states, there are more stringent requirements in terms of driving experience to be able to take a road test and more restrictions on what a younger driver can do with his/her license. The license simply isn’t as valuable as it used to be.
  • Driving has peaked in the US. People are driving less frequently and fewer miles when they do. There has also been a movement of the population to urban areas which have more mass transit.
  • The decline of retail has played a part. Going to the mall was a common weekend activity for Xer teens. Now, staying home and shopping on Amazon is more common. Millennials never went to the mall to socialize.
  • Online entertainment options have proliferated. Movies and shows are readily streamed. Many teens fulfill a need for socialization via gaming, where they interact with their friends and make new ones. This need could only be met in person in the past.
  • Teens are working less so have less of a need to drive to work. Of course, this means they have less of their own money and that tethers them to their parents even longer.

There are likely many other causes. But the result is clear. Teens are getting licenses later and using them less than they did a generation ago.

As a result, researchers have lost a perfectly good measure! Obtaining a driver’s license is not as strong a rite of passage as it used to be.

We’ve been thinking about what might make a good alternative measure. What life event do young people experience that changes them in terms of granting their independence from parents? Leaving home and living independently for the first time would qualify but seems a bit late to be useful. There may be no clear marker signifying independence for Millennials, as they stay dependent on parents across a much wider time period than in the past. Or, perhaps we need to change our definition of independence.

Why Lori is the Best Shark in the Tank

Shark Tank is one of my favorite TV shows. It showcases aspiring entrepreneurs as they make business presentations to an investor panel, who then choose whether to invest. It is fun to play along and try to predict how the Sharks will react to a business pitch. As a small business owner/entrepreneur, it is fun to imagine how I might do in a Shark Tank presentation. And, it was interesting to watch a college teammate of mine make a successful pitch on the show.

My inner geek came out when watching a recent episode. I got to wondering how much the need to entertain might cloud how the venture capital world is portrayed on the show. How many Shark Tank pitches actually result in successful companies? Is the success rate for Shark Tank businesses any higher than any other small company looking for growth capital? Are there any biases in the way the Shark Tanks choose to invest?

This curiosity led to a wasted work day.

Venture capital, especially at early stages, involves high risk bets. Firms may invest in 100 companies knowing full well that 80 or 90 of them will fail, but that a handful of wild successes will pay off handsomely. It isn’t for the faint of heart. I found an interview with Mark Cuban where he stated he hoped that 15% of his Shark Tank investments would eventually pay off. Even that seems high. Given that he has invested about $32 million so far, that is an admission that $27.5 million of that is expected to be wasted. Gutsy.

I also was able to discover interesting things about the show that are largely hidden from the viewer:

  1. The Sharks themselves are paid to take part. I was able to find discussions that suggested they may make as much as $100K per episode. That is a million dollars or more per season, so perhaps they are playing with house money more than they let on.
  2. Getting on Shark Tank is statistically harder than getting into an Ivy League college. It is estimated that more than 50,000 people apply for each season with less than 1% being successful. That alone should provide some realism as to the probability of success of new businesses.
  3. In the early seasons, an entrepreneur had to give up 2% of revenue or 5% of his/her company to the production firm just to appear on the show. That requirement was removed in later seasons because Mark Cuban refused to remain on the show if it remained.
  4. Many of the deals you see made on the show don’t end up being consummated. Forbes conducted survey research in 2016 that indicated that 43% of Shark Tank deals fell apart in the due diligence stage and 30% of the time the deal changed substantially from what is seen on TV. The deal you see on TV only came to fruition as you saw it about 1 in 4 times (27% of the time).

This makes it challenging to assess the deals and whether or not they paid off. Shark Tank companies are almost all privately-held so their revenue data is tough to come by and we can’t really know for sure what the deal was.

Although we can’t review business outcomes as we might like, we can look closely at the deals themselves. The data we used for this includes all deals and prospective deals from the first nine seasons of the show. So, it does not include the current season, which premiered in October 2018.

In the first nine seasons, there were 803 pitches resulting in 436 closed on-air deals (53% of pitches). Applying the Forbes data would imply that of these 436 deals, 187 of them likely feel apart, and 131 of them likely changed substantially. The net? Our projection would be that 53% of pitches result in handshakes on-air, but post-show only 37% of all original pitches close at all and only 15% of pitches will close at the terms you see on air.

Why would Shark Tank deals fail to close? There is a due diligence stage where investors get to have their accountants review the entrepreneur’s books. I found some articles that indicated that some entrepreneurs got cold feet and refused the deal after the show. Also, some of the deals have contingencies which fail to occur.

It is interesting to look at deals by the gender of the entrepreneur as it shows that Shark Tank entrepreneurs skew heavily male:

  • Men are much more likely than women to appear as entrepreneurs on Shark Tank. Of the 803 pitches, 482 (60%) made by men, 198 (25%) by women, and 119 (15%) by mixed teams of men and women. So, 75% of the time, at least one male is involved in the pitch, and 40% of the time at least one female is involved in the pitch.
  • However, women (59% closed) are more likely than men (51%) to successfully close a deal on air.

There are data that imply that men and women negotiate differently:

  • Men initially ask for higher company valuations ($4.5 million on average) than women ($3.1 million on average).
  • Men also ask for more capital ($342K on average) than women ($238K on average).
  • Men (47%) and women (49%) receive about the same proportion of their initial valuation ask. Men (94%) and women (88%) also receive about the same proportion of cash that they initially ask for.

So, men are far more likely to appear on the show and come with bigger deals on average than women. But they receive (proportionately) about the same discount on their deals as women as they negotiate with the Sharks. If there is a difference in their negotiation skills it is that men start bigger or come to the show with more mature companies.

We can also look at individual Sharks to get a sense of how good of negotiators they are:

  • Mark is the most aggressive Shark. He has the most deals (132, despite not being on the early seasons of the show) as well as the most invested (about $32 million).
  • The cheapest (or most frugal?) Sharks are Barbara and Daymond. Barbara has put forth the least amount of money (about $10 million) and her average deal valuation is $945K. Daymond has put out the second least amount of money (about $12 million) and has an average deal size of $957K. These two Sharks have likely not put much more money into their Shark investments than they have been paid to be on the show.
  • Mr. Wonderful seems to have a “go big or stay home” mentality. He has closed the fewest deals (64) of any Shark. But, his average deal valuation of $2.7 million is the highest of any Shark.
  • Lori and Kevin (31% of pitches) are the most likely to make an offer. Barbara and Daymond (22%) are least likely to make an offer.
  • So, Kevin make the most offers and closes the fewest deals, making him the least desirable Shark from the standpoint of the entrepreneurs.

Barbara is the most likely to invest in a female entrepreneur. She is about as likely to invest in a female entrepreneur as a male entrepreneur despite the fact that so many more men than women appear on the show. Kevin and Robert are the least likely to invest in a female entrepreneur. Mark and Daymond demonstrate no bias, as the invest in about the same proportion as appearances on the show.

ALL

Barbara Lori Mark Daymond Kevin

Robert

Male

60%

44% 53% 60% 57% 67%

71%

Female

25%

42% 33% 27% 25% 19%

17%

Mixed Team

15%

14% 14% 13% 18% 14%

12%

So, who has been the most successful Shark? It can be hard to tell because data are scarce, but my vote would go for Lori. USA Today put out a list of the top 20 best-selling products featured on Shark Tank. Six of the top 10 were from investments Lori made, including the top 3. Eight of the top 10 investments by revenue were made by the two female Sharks, Lori and Barbara.

Who are the worst Sharks in terms of revenue performance? My vote here would be a tie between Mark and Daymond. Mark has just 3 of the top 20 investments and Daymond has just 2. If we can assume that the goal of venture capital is to generate big wins, it is clear that Lori and Barbara are killing it and Mark and Daymond are not.

Shark Tank is a great catalyst for entrepreneurs, but because it is entertainment and not reality it can mischaracterize entrepreneurship in the real world. Sharks may invest for the entertainment value of the show and because investing boosts their personal brand as much as the product. And, it might just be the case that the amount of money they have invested is not much larger than the amount of money they have been paid to be on the show.

Almost all successful people will tell you that learning from their failures was at least as important as their successes, yet Shark Tank never revisits failed investments and it is likely that the bulk of the deals we see on TV do not end up paying off for the investor. The show does not disclose how few of its deals actually come to fruition once the cameras are no longer rolling. Just once I’d like to see an update segment show an investment that failed miserably.

Shark Tank also seems to imply that hard work and grit always triumph, when in reality knowing when to cut losses and having a little bit of luck matters a lot in business success. Grit matters for sure, but not when it’s focus is blind and irrational, and it can be sad to see entrepreneurs who have sacrificed so much and it is clear their business is not going to make it.

At its best, Shark Tank stimulates people to think like an entrepreneur. At its worst, it presents too rosy a picture of small business life which influences people to invent new products and launch companies that are likely to fail, at great consequence to the entrepreneur. It certainly provides great entertainment.

How Did Pollsters Do in the Midterm Elections?

Our most read blog post was posted the morning after the 2016 Presidential election. It is a post we are proud of because it was composed in the haze of a shocking election result. While many were celebrating their side’s victory or in shock over their side’s losses, we mused about what the election result meant for the market research industry.

We predicted pollsters would become defensive and try to convince everyone that the polls really weren’t all that bad. In fact, the 2016 polls really weren’t. Predictions of the popular vote tended to be within a percent and a half or so of the actual result which was better than for the previous Presidential election in 2012. However, the concern we had about the 2016 polls wasn’t related to how close they were to the result. The issue we had was one of bias: 22 of the 25 final polls we found made an inaccurate prediction and almost every poll was off in the same direction. That is the very definition of bias in market research.

Suppose that you had 25 people flip a coin 100 times. On average, you’d expect 50% of the flips to be “heads.” But, if say, 48% of them were “heads” you shouldn’t be all that worried as that can happen. But, if 22 of the 25 people all had less than 50% heads you should worry that there was something wrong with the coins or they way they were flipped. That is, in essence, what happened in the 2016 election with the polls.

Anyway, this post is being composed the aftermath of the 2018 midterm elections. How did the pollsters do this time?

Let’s start with FiveThirtyEight.com. We like this site because they place probabilities around their predictions. Of course, this gives them plausible deniability when their prediction is incorrect, as probabilities are never 0% or 100%. (In 2016 they gave Donald Trump a 17% chance of winning and then defended their prediction.) But this organization looks at statistics in the right way.

Below is their final forecast and the actual result. Some results are still pending, but at this moment, this is how it shapes up.

  • Prediction: Republicans having 52 seats in the Senate. Result: It looks like Republicans will have 53 seats.
  • Prediction: Democrats holding 234 and Republicans holding 231 House seats. Result: It looks like Democrats will have 235 or 236 seats.
  • Prediction: Republicans holding 26 and Democrats holding 24 Governorships. Result: Republicans now hold 26 and Democrats hold 24 Governorships.

It looks like FiveThirtyEight.com nailed this one. We also reviewed a prediction market and state-level polls, and it seems that this time around the polls did a much better job in terms of making accurate predictions. (We must say that on election night, FiveThirtyEight’s predictions were all over the place when they were reporting in real time. But, as results settled, their pre-election forecast looked very good.)

So, why did polls seem to do so much better in 2018 than 2016? One reason is the errors cancel out when you look at large numbers of races. Sure, the polls predicted Democrats would have 234 seats, and that is roughly what they achieved. But, in how many of the 435 races did the polls make the right prediction? That is the relevant question, as it could be the case that the polls made a lot of bad predictions that compensated for each other in the total.

That is a challenging analysis to do because some races had a lot of polling, others did not, and some polls are more credible than others. A cursory look at the polls suggests that 2018 was a comeback victory for the pollsters. We did sense a bit of an over-prediction favoring the Republican Senatorial candidates, but on the House side there does not seem to be a clear bias.

So, what did the pollsters do differently? Not much really. Online sampling continues to evolve and get better, and the 2016 result has caused polling firms to concentrate more carefully on their sampling. One of the issues that may have caused the 2016 problem is that pollsters are starting to almost exclusively use the top 2 or 3 panel companies. Since 2016, there has been a consolidation among sample suppliers, and as a result, we are seeing less variance in polls as pollsters are largely all using the same sample sources. The same few companies provide virtually all the sample used by pollsters.

Another key difference was that turnout in the midterms was historically high. Polls are more accurate in high turnout races, as polls almost always survey many people who do not end up showing up on election day, particularly young people. However, there are large and growing demographic differences (age, gender, race/ethnicity) in supporters of each party, and that greatly complicates polling accuracy. Some demographic subgroups are far more likely than others to take part in a poll.

Pollsters are starting to get online polling right. A lot of the legacy firms in this space are still entrenched in the telephone polling world, have been protective of their aging methodologies, and have been slow to change. After nearly 20 years of online polling the upstarts have finally forced the bigger polling firms to question their approaches and to move to a world where telephone polling just doesn’t make a lot of sense. Also, many of the old guard, telephone polling experts are now retired or have passed on, and they have largely led the resistance to online polling.

Gerrymandering helps the pollster as well. It still remains the case that relatively few districts are competitive. Pew suggests that only 1 in 7 districts was competitive. You don’t have to be a pollster to accurately predict how about 85% of the races will turn out. Only about 65 of the 435 house races were truly at stake. If you just flipped a coin in those races, in total your prediction of house seats would have been fairly close.

Of course, pollsters may have just gotten lucky. We view that as unlikely though, as there were too many races. Unlike in 2018 though, in 2016 we haven’t seen any evidence of bias (in a statistical sense) in the direction of polling errors.

So, this is a good comeback success for the polling industry and should give us greater confidence for 2020. It is important that the research industry broadcasts this success. When pollsters have a bad day, like they did in 2016, it affects market research as well. Our clients lose confidence in our ability to provide accurate information. When the pollsters get it right, it helps the research industry as well.

The most selective colleges have the least effective marketing

Recently, Stanford University made headlines for deciding to stop issuing an annual press release documenting its number of applicants and acceptances.

There has been a bit of an arms race among colleges with competitive admissions to be able to claim just how selective they are. The smaller the proportion of applicants accepted, the better the college does in many ranking systems and the more exclusive the “brand” of the college becomes.

This seems to be a bit crazy, as publicizing how few students are accepted is basically broadcasting how inefficient your college marketing system has become. We can’t think of any organization beyond colleges that would even consider doing something analogous to this – broadcasting to the world that they have enticed non-qualified buyers to consider their product.

I learned firsthand how ingrained this behavior is among college admissions and marketing personnel. About five years ago I had the pleasure to speak in front of a group of about 200 college marketers and high school counselors. I created what I felt was a compelling and original talk which took on this issue. I have given perhaps 200 talks in my career, and this one might have been the single most poorly received presentation I have ever delivered.

The main thrust of my argument was that as a marketer, you want to be as targeted as possible so as to not waste resources. “Acquisition cost” is an important success metric for markers: how much do you spend in marketing for every customer you are able to obtain? Efficiency in obtaining customers is what effective marketing is all about.

I polled the audience to ask what they felt the ideal acceptance rate would be for their applicants. Almost all responded “under 10%” and most responded “under 5%.” I then stated that the ideal acceptance rate for applicants would be 100%. The ideal scenario would be this: every applicant to your college would be accepted, would then choose to attend your institution, would go on to graduate, become a success, and morph into an engaged alumnus.

I used an analogy of a car dealership. Incenting college marketers to increase applications is akin to compensating a car salesperson for how many test drives he/she takes customers on. The dealership derives no direct value from a test drive. Every test drive that does not result in a car purchase is a waste of resources. The test drive is a means to an end and car dealers don’t tend to track it as a success metric. Instead, they focus on what matters – how many cars are sold and how much was spent in marketing to make that happen.

Colleges reward their marketers to get students to test drive when they should be rewarding their marketers for getting them to buy. This wouldn’t matter much if a high proportion of applicants were accepted and ending up attending.  But, even at highly selective colleges it is not uncommon for less than 10% of applicants to be accepted, less than 33% of those accepted to choose to attend, and less than 50% of those that enroll to actually end up graduating. At those rates, for every 1,000 applicants, just 17 will end up graduating from the institution. That is a success rate of 1.7%.

These are metrics that in any business context would be seen as a sign of an organization in serious trouble. Can you imagine if only 10% of the people who came in your store qualified to buy your product? And then if only a third of those would actually decide to do so? And then if half of those that do buy don’t end up using your product or return it? That is pretty much what happens at selective colleges.

This issue is a failure of leadership. College marketers I have worked with can often see this problem, but feel pressured by their Deans and College Presidents to maximize their applicant base. Granted, this can help build the college’s brand, but it is a huge drain on resources that are better spent ensuring targeting applicants who are poised for success at the institution. It has happened because selectivity is considered important in building a college’s brand. Stanford has taken a useful first step, and hopefully other colleges will follow their lead.

Is segmentation just discrimination with an acceptable name?

A short time ago we posted a basic explanation of the Cambridge Analytica/Facebook scandal (which you can read here). In it, we stated that market segmentation and stereotyping are essentially the same thing. This presents an ethical quandary for marketers as almost every marketing organization makes heavy use of market segmentation.

To review, marketers place customers into segments so that they can better understand and serve them. Segmentation is at the essence of marketing. Segments can be created along any measurable dimension, but since almost all segments have a demographic component we will focus on that for this post.

It can be argued that segmentation and stereotyping are the same thing. Stereotyping is attaching perceived group characteristic to an individual. For instance, if you are older I might assume your political views lean conservative, since it is known that political views tend to be more conservative in older Americans that they are in general among younger Americans. If you are female I might assume you are more likely to be the primary shopper for your household, since females in total do more of the family shopping than males. If you are African-American, I might assume you have a higher likelihood than others to listen to rap music, since that genre indexes high among African-Americans.

These are all stereotypes. These examples can be shown to true of a larger group, but that doesn’t necessarily imply that they apply to all the individuals in the group. There are plenty of liberal older Americans, females who don’t shop at all, and African-Americans who can’t stand rap music.

Segmenting consumers (which is applying stereotypes) isn’t inherently a bad thing. It leads to customized products and better customer experiences. The potential problem isn’t with stereotyping, it is when doing so moves to a realm of being discriminatory that we have to be careful. As marketers we tread a fine line. Stereotyping oversimplifies the complexity of consumers by forming an easy to understand story. This is useful in some contexts and discriminatory in others.

Some examples are helpful. It can be shown that African-Americans have a lower life expectancy than Whites. A life insurance company could use this information to charge African-Americans higher premiums than Whites. (Indeed, many insurance companies used to do this until various court cases prevented them from doing so.) This is a segmentation practice that many would say crosses a line to become discriminatory.

In a similar vein, car insurance companies routinely charge higher risk groups (for example younger drivers and males) higher rates than others. That practice has held up as not being discriminatory from a legal standpoint, largely because the discrimination is not against a traditionally disaffected group.

At Crux, we work with college marketers to help them make better admissions offer decisions. Many colleges will document the characteristics of their admitted students who thrive and graduate in good standing. The goal is to profile these students and then look back at how they profiled as applicants. The resulting model can be used to make future admissions decisions. Prospective student segments are established that have high probabilities of success at the institution because they look like students known to be successful, and this knowledge is used to make informed admissions offer decisions.

However, this is a case where a segmentation can cross a line and become discriminatory. Suppose that the students who succeed at the institution tend to be rich, white, female, and from high performing high schools. By benchmarking future admissions offers against them, an algorithmic bias is created. Fewer minorities, males, and students from urban districts will be extended admissions offers What turns out to be a good model from a business standpoint ends up perpetuating a bias., and places certain demographics of students at a further disadvantage.

There is a burgeoning field in research known as “predictive analytics.” It allows data jockeys to use past data and artificial intelligence to make predictions on how consumers will react. It is currently mostly being used in media buying. Our view is it helps in media efficiency, but only if the future world can be counted on to behave like the past. Over-reliance on predictive analytics will result in marketers missing truly breakthrough trends. We don’t have to look further than the 2016 election to see how it can fail; many pollsters were basing their modeling on how voters had performed in the past and in the process missed a fundamental shift in voter behavior and made some very poor predictions.

That is perhaps an extreme case, but shows that segmentations can have unintended consequences. This can happen in consumer product marketing as well. Targeted advertising can become formulaic. Brands can decline distribution in certain outlets. Ultimately, the business can suffer and miss out on new trends.

Academics (most notably Kahneman and Tversky) have established that people naturally apply heuristics to decision making. These are “rules of thumb” that are often useful because they allow us to make decisions quickly. However, these academics have also demonstrated how the use of heuristics often result in sub-optimal and biased decision making.

This thinking applies to segmentation. Segmentation allows us to make marketing decisions quickly because we assume that individuals take on the characteristics of a larger group. But, it ignores the individual variability within the group, and often that is where the true marketing insight lies.

We see this all the time in the generational work we do. Yes, Millennials as a group tend to be a bit sheltered, yet confident and team-oriented. But this does not mean all of them fit the stereotype. In fact, odds are high that if you profile an individual from the Millennial generation, he/she will only exhibit a few of the characteristics commonly attributed to the generation. Taking the stereotype too literally can lead to poor decisions.

This is not to say that marketers shouldn’t segment their customers. This is a widespread practice that clearly leads to business results. But, they should do so considering the errors and biases applying segments can create, and think hard about whether this can unintentionally discriminate and, ultimately, harm the business in the long term.

Has market research become Big Brother?

Technological progress has disrupted market research. Data are available faster and cheaper than ever before. Many traditional research functions have been automated out of existence or have changed significantly because of technology. Projects take half the time to complete that they did just a decade ago. Decision making has moved from an art to a science. Yet, as with most technological disruptions, there are just as many potential pitfalls as efficiencies to be wary of as technology changes market research.

“Passive” data collection is one of these potential pitfalls. It is used by marketers in good ways: the use of passive data helps understand consumers better, target meaningful products and services, and create value for both the consumer and the marketer. However, much of what is happening with passive data collection is done without the full knowledge of the consumer and this process has the potential of being manipulative. The likelihood of backlash towards the research industry is high.

The use of passive data in marketing and research is new and many researchers may not know what is happening so let us explain. A common way to obtain survey research respondents is to tap into large, opt-in online panels that have been developed by a handful of companies. These panels are often augmented with social (river) channels whereby respondents are intercepted while taking part in various online activities. A recruitment email or text is delivered, respondents take a survey, and data are analyzed. Respondents provide information actively and with full consent.

There have been recent mergers which have resulted in fewer but larger and more robust online research panels available. This has made it feasible for some panel companies to gain the scale necessary to augment this active approach with passive data.

It is possible to append information from all sorts of sources to an online panel database. For instance, voter registration files are commonly appended. If you are in one of these research panels, clients likely know if you are registered to vote, if you actually voted, and your political party association. They will have made a prediction of how strong a liberal or conservative you likely are. They may have even run models to predict which issues you care most about. You are likely linked into a PRIZM cluster that associates you with characteristics of the neighborhood where you reside, which in turn can score your potential to be interested in all sorts of product categories. This is all in your file.

These panels also have the potential to link to other publicly-available databases such as car registration files, arrest records, real estate transactions, etc. If you are in these panels, whether you have recently bought a house, how much you paid for it, if you have been convicted of a crime, may all be in your “secret file.”

But, it doesn’t stop there. These panels are now cross-referenced to other consumer databases. There are databases that gather the breadcrumbs you leave behind in your digital life: sites you are visiting, ads you have been served, and even social media posts you have made. There is a tapestry of information available that is far more detailed than most consumers realize. From the research panel company’s perspective, it is just a matter of linking that information to their panel.

This opens up exciting research possibilities. We can now conduct a study among people who are verified to have been served by a specific client’s digital advertising. We can refine our respondent base further by those who are known to have clicked on the ad. As you can imagine, this can take ad effectiveness research to an entirely different level. It is especially interesting to clients because it can help optimize media spending which is by far the largest budget item for most marketing departments.

But, therein lies the ethical problem. Respondents, regardless of what privacy policies they may have agreed to, are unlikely to know that their passive web behavior is being linked into their survey responses. This alone should ring alarm bells for an industry suffering from low response rates and poor data quality. Respondents are bound to push back when they realize there is a secret file panel companies are holding on them.

Panel companies are straying from research into marketing. They are starting to encourage clients to use the survey results to better target individual respondents in direct marketing. This process can close a loop with a media plan. So, say on a survey you report that you prefer a certain brand of a product. That can now get back to you and you’ll start seeing ads for that product, likely without your knowledge that this is happening because you took part in a survey.

To go even further, this can affect advertising people not involved in the survey may see. If you prefer a certain brand and I profile a lot like you, as a result of your participation in a survey I may end up seeing specific ads. Even if I don’t know you or have any connection to you.

In some ways, this reeks of the Cambridge Analytica scandal (which we explain in a blog post here). We’ll be surprised if this practice doesn’t eventually create a controversy in the survey research industry. This sort of sales targeting resulting from survey participation will result in lower response rates and a further erosion of confidence in the market research field. However, it is also clear that these approaches are inevitable and will be used more and more as panel companies and clients gain experience with them.

It is the blurring of the line between marketing and market research that has many old-time researchers nervous. There is a longstanding ethical tenet in the industry that participation in research project should in no way result in the respondent being sold or marketed to. The term for this is SUGGING (Selling Under the Guise of research) and all research industry trade groups have a prohibition against SUGGING embedded in their codes of ethics. It appears that some research firms are ignoring this. But, this concept has always been central to the market research field: we have traditionally assured respondents that they can be honest on our surveys because we will in no way market to them directly because of their answers.

In the novel 1984 George Orwell describes a world where the government places its entire civilization under video surveillance. For most of the time since its publication, this has appeared as a frightening, far-fetched cautionary tale. Recent history has suggested this world may be upon us. The NSA scandal (precipitated by Edward Snowden) showed how much of our passive information is being shared with the government without our knowledge. Rather than wait for the government to surveil the population, we’ve turned the cameras on ourselves. Marketers can do things I don’t feel people realize and research respondents are unknowingly enabling this. The contrails you leave as you simply navigate your life online can be used to follow you and the line between research and marketing is fading, and this will eventually be to the detriment of our field.

Sexual harassment/abuse among college students – new survey results released

Sexual harassment and abuse on college campuses has garnered increased attention in the media and by political leaders. Surprisingly, there is little research documenting what is actually happening among college students – what the levels of abuse and harassment are, who is being victimized, and how students feel their college administrators are dealing with these issues.

In the spring of 2018 Crux Research surveyed 717 current college students to learn more about the current state of these issues. An issue like sexual harassment can be challenging to get right from a polling standpoint because it can be difficult to define. As a general term, it can be too broad to interpret as different experiences may be construed by one person as harassment and as another as not being harassment. The best way to address this is to be specific in our questioning. To be sure respondents understood our objectives, we developed a list of statements under three harassment categories shown below:

Verbal/Non-Physical harassment

  • Being called gay or lesbian in a negative way
  • Being shown sexy or sexual pictures you didn’t want to see
  • Being verbally intimidated in a sexual way
  • Having someone make unwelcome sexual comments, jokes, or gestures to or about you
  • Having someone flash or expose themselves to you

Online harassment

  • Being called gay or lesbian in a negative way online
  • Having someone spread unwelcome sexual rumors about you online
  • Having someone post unwelcome sexual comments, jokes, or pictures about or of you online
  • Being sent unwelcome sexual comments, jokes, or pictures electronically

Physical harassment

  • Being physically intimidated in a sexual way
  • Being touched in an unwelcome sexual way
  • Being forced to do something sexual you didn’t want to do

For each, we asked the college student if he/she had been a victim of the specific type of harassment since they had been a college student. We found that 54% of college students have been a victim of some form of verbal/non-physical harassment, 45% have been a victim of some sort of online harassment, and 32% have been a victim of some sort of physical harassment.

Importantly, this study finds that while victimization is usually thought of as an issue for college women, college men are also common victims of sexual harassment:

  • 55% of college females have been the victims of verbal harassment, compared to 52% of college males.
  • 42% of college females have been the victims of online harassment, compared to 47% of college males.
  • 32% of college females have been the victims of physical harassment, compared to 32% of college males.

There are some large differences in college males and females, depending on the specific form of harassment:

College females are more likely than college males to report that…

  • Someone has made unwelcome sexual comments, jokes, or gestures to or about them (41% of females; 17% of males).
  • They have been verbally intimidated in a sexual way (27% of females; 17% of males).
  • They have been sent unwelcome sexual comments, jokes, or pictures electronically (30% of females; 17% of males).

College males are more likely than college females to report that…

  • Being called gay or lesbian in a negative way (20% of males; 14% of females).
  • Being called gay or lesbian in a negative way online (20% of males; 8% of females).

Perhaps most surprising is that for the most serious abuse item presented (“being forced to do something sexual that you didn’t want to do”) there was no statistical difference between college males and college females. Overall, 13% of college students indicated this has happened to them since they have been at college – about 1 in 8 college students. Again, the most serious types of sexual harassment and abuse happening on campuses is not solely a female issue. College men are reporting being sexual abused in a physical way as well.

Although we have shown that victimization is not solely an issue for college females, it is clear from our study that the perpetrators of sexual harassment/abuse are predominantly male. Overall, victims report that 72% of the time their harasser was male, 16% of the time the harasser was female, and 12% of the time it was both.

Most commonly, victims report that their harasser was a fellow college student (53%) or a friend (26%). 12% report that their harasser was a romantic partner. It is rare that students will report that their instructors/professors (4% of cases) or another adult at college (3%) are the harassers. Sexual harassment on college campuses appears to be mostly peer-to-peer.

Unique to this study, we also asked college students if they had done anything since they had been a student that could be correctly interpreted as being sexual harassment. Seventeen percent (17%) of students said they had – including 28% of all college males. To repeat: more than one in four (28%) of college males admit that they have done something to sexually harass another student since they have been in college.

Perhaps most troubling is how infrequently instances of abuse are reported. This study indicates that just 37% of harassment gets reported. Females (reporting 24% of instances) are less likely than males (54%) to make a report. For every report made by a college female, there are three incidents that are not reported.  And, our study also found that instances where the harasser was a fellow student are the ones that are least likely to be reported.

This issue has been brought more front and center at colleges in the past few years. College culture is moving towards supporting the victim/accuser. Compared to a year ago, about half (52%) of students are more likely to believe someone that reports being sexually harassed and 15% are less likely to believe someone who reports harassment. About two-thirds (65%) of students think the greater focus on these issues will result in a long-term change in attitudes about sexual harassment at college. Three-quarters (74%) feel that unreported sexual harassment is a bigger issue than false reporting of sexual harassment.

College students are largely satisfied with how their administration has addressed sexual misconduct and harassment. Overall, just 6% felt that their administration is not taking this issue seriously. Seventy percent (70%) feel that their college provides enough protection against sexual harassment and abuse.

In sum, sexual harassment and abuse occurs at a troubling level at colleges – and both college females and males are victims. Students are rallying behind the accusers, yet far too few victims are reporting harassment incidences, especially when they happen student-to-student. It appears that students have confidence in their administrators to handle these issues and protect them.

Market research isn’t about storytelling, it is about predicting the future

We recently had a situation that made me question the credibility of market research. We had fielded a study for a long-term client and were excited to view the initial version of the tabs. As we looked at results by age groupings we found them to be surprising. But this was also exciting because we were able to weave a compelling narrative around why the age results seemed counter-intuitive.

Then our programmer called to say a mistake had been made in the tabs and the banner points by age had been mistakenly reversed.

So, we went back to the drawing board ad constructed another, equally compelling story, as to why the data were behaving as they were.

This made me question the value of research. Good researchers can review seemingly disparate data points from a study and generate a persuasive story as to why they are as they are. Our entire business is based on this skill – in the end clients pay us to use data to provide insight into their marketing issues. Everything else we do is a means to this end.

Our experience with the flipped age banner points illustrates that stories can be created around any data. In fact, I’d bet that if you gave us a randomly-generated data set we could convince you as to its relevance to your marketing issues. I actually thought about doing this – taking the data we obtain by running random data through a questionnaire when testing it before fielding, handing it to an analyst, and seeing what happens. I’m convinced we could show you a random data set’s relevance to your business.

This issue is at the core of polling’s PR problem. We’ve all heard people say that you can make statistics say anything, therefore polls can’t be trusted. There are lies, damn lies, and statistics. I’ve argued against this for a long time because the pollsters and researchers I have known have universally been well-intentioned and objective and never try to draw a pre-determined conclusion from the data.

Of course, this does not mean that all of the stories we tell with data aren’t correct or enlightening. But, they all come from a perspective. Clients value external suppliers because of this perspective – we are third-party observers who aren’t wrapped up in the internal issues client’s face and we are often in a good position to view data with an objective mind. We’ve worked with hundreds of organizations and can bring these experiences bring that to bear on your study. Our perspective is valuable.

But, it is this perspective that creates an implicit bias in all we do. You will assess a data set from a different set of life experiences and background than I will. That is just human nature. Like all biases in research, our implicit bias may or not be relevant to a project. In most cases, I’d say it likely isn’t.

So, how can researchers reconcile this issue and sleep at night knowing their careers haven’t been a sham?

First and foremost, we need to stop saying that research is all about storytelling. It isn’t. The value of market research isn’t in the storytelling it is in the predictions of the future it makes. Clients aren’t paying us to tell them stories. They are paying us to predict the future and recommend actions that will enhance their business. Compelling storytelling is a means to this but is not our end goal. Data-based storytelling provides credibility to our predictions and gives confidence that they have a high probability of being correct.

In some sense, it isn’t the storytelling that matters, it is the quality of the prediction. I remember having a college professor lecturing on this. He would say that the quality of a model is judged solely by its predictive value. Its assumptions, arguments, and underpinnings really didn’t matter.

So, how do we deal with this issue … how do we ensure that the stories we tell with data are accurate and fuel confident predictions? Below are some ideas.

  1. Make predictions that can be validated at a later date. Provide a level of confidence or uncertainty around the prediction. Explain what could happen to prevent your prediction from coming true.
  2. Empathize with other perspectives when analyzing data. One of the best “tricks” I’ve ever seen is to re-write a research report as if you were writing it for your client’s top competitor. What conclusions would you draw for them? If it is an issue-based study, consider what you would conclude from the data if your client was on the opposite side of the issue.
  3. Peg all conclusions to specific data points in the study. Straying from the data is where your implicit bias may tend to take over. Being able to tie conclusions directly to data is dependent on solid questionnaire design.
  4. Have a second analyst review your work and play devil’s advocate. Show him/her the data without your analysis and see what stories and predictions he/she can develop independent of you. Have this same person review your story and conclusions and ask him/her to try to knock holes in them. The result is a strengthened argument.
  5. Slow down. It just isn’t possible to provide stories, conclusions, and predictions from research data that consider differing perspectives when you have just a couple of days to do it. This requires more negotiation upfront as to project timelines. The ever-decreasing timeframes for projects are making it difficult to have the time needed to objectively look at data.
  6. Realize that sometimes a story just isn’t there. Your perspective and knowledge of a client’s business should result in a story leaping out at you and telling itself. If this doesn’t happen, it could be because the study wasn’t designed well or perhaps there simply isn’t a story to be told. The world can be a more random place than we like to admit, and not everything you see in a data set is explainable. Don’t force it – developing a narrative that is reaching for explanations is inaccurate and a disservice to your client.

Visit the Crux Research Website www.cruxresearch.com

Enter your email address to follow this blog and receive notifications of new posts by email.