Archive for the 'Uncategorized' Category

Among college students, Bernie Sanders is the overwhelming choice for the Democratic nomination

Crux Research poll of college students shows Sanders at 23%, Biden at 16%, and all other candidates under 10%

ROCHESTER, NY – October 10, 2019 – Polling results released today by Crux Research show that if it was up to college students, Bernie Sanders would win the Democratic nomination the US Presidency. Sanders is the favored candidate for the nomination among 23% of college students compared to 16% for Joe Biden. Elizabeth Warren is favored by 8% of college students followed by 7% support for Andrew Yang.

  • Bernie Sanders: 23%
  • Joe Biden: 16%
  • Elizabeth Warren: 8%
  • Andrew Yang: 7%
  • Kamala Harris: 6%
  • Beto O’Rourke: 5%
  • Pete Buttigieg: 4%
  • Tom Steyer: 3%
  • Cory Booker: 3%
  • Michael Bennet: 2%
  • Tulsi Gabbard: 2%
  • Amy Klobuchar: 2%
  • Julian Castro: 1%
  • None of these: 5%
  • Unsure: 10%
  • I won’t vote: 4%

The poll also presented five head-to-head match-ups. Each match-up suggests that the Democratic candidate currently has a strong edge over President Trump, with Sanders having the largest edge.

  • Sanders versus Trump: 61% Sanders; 17% Trump; 12% Someone Else; 7% Not Sure; 3% would not vote
  • Warren versus Trump: 53% Warren; 18% Trump; 15% Someone Else; 9% Not Sure; 5% would not vote
  • Biden versus Trump: 51% Biden; 18% Trump; 19% Someone Else; 8% Not Sure; 4% would not vote
  • Harris versus Trump: 48% Harris; 18% Trump; 20% Someone Else; 10% Not Sure; 4% would not vote
  • Buttigieg versus Trump: 44% Buttigieg; 18% Trump; 22% Someone Else; 11% Not Sure; 5% would not vote

The 2020 election could very well be determined on the voter turnout among young people, which has traditionally been much lower than among older age groups.

###

Methodology
This poll was conducted online between October 1 and October 8, 2019. The sample size was 555 US college students (aged 18 to 29). Quota sampling and weighting were employed to ensure that respondent proportions for age group, sex, race/ethnicity, and region matched their actual proportions in the US college student population.

This poll did not have a sponsor and was conducted and funded by Crux Research, an independent market research firm that is not in any way associated with political parties, candidates, or the media.

All surveys and polls are subject to many sources of error. The term “margin of error” is misleading for online polls, which are not based on a probability sample which is a requirement for margin of error calculations. If this study did use probability sampling, the margin of error would be +/-4%.

About Crux Research Inc.
Crux Research partners with clients to develop winning products and services, build powerful brands, create engaging marketing strategies, enhance customer satisfaction and loyalty, improve products and services, and get the most out of their advertising.

Using quantitative and qualitative methods, Crux connects organizations with their customers in a wide range of industries, including health care, education, consumer goods, financial services, media and advertising, automotive, technology, retail, business-to-business, and non-profits.
Crux connects decision makers with customers, uses data to inspire new thinking, and assures clients they are being served by experienced, senior level researchers who set the standard for customer service from a survey research and polling consultant.

To learn more about Crux Research, visit http://www.cruxresearch.com.

How to be an intelligent consumer of political polls

As the days get shorter and the air gets cooler, we are on the edge of a cool, colorful season. We are not talking about autumn — instead, “polling season” is upon us! As the US Presidential race heats up, one thing we can count on is being inundated with polls and pundits spinning polling results.

Most market researchers are interested in polls. Political polling pre-dates the modern market research industry and most market research techniques used today have antecedents from the polling world. And, as we have stated in a previous post, polls can be as important as the election itself.

The polls themselves influence voting behavior which should place polling organizations in an ethical quandary. Our view is that polls, when properly done, are an important facet of modern democracy. Polls can inform our leaders as to what the electorate cares about and keep them accountable. This season, polls are determining which candidates get on the debate stage and are driving which issues candidates are discussing most prominently.

The sheer number of polls that we are about to see will be overwhelming. Some will be well-conducted, some will be shams, and many will be in between. To help, we thought we’d write this post on how be an intelligent consumer of polls and what to look out for when reading the polls or hearing about them in the media.

  • First, and this is harder than it sounds, you have to put your own biases aside. Maybe you are a staunch conservative or liberal or maybe you are in the middle. Whatever your leaning, your political views are likely going to get in the way of you becoming a good reader of the polls. It is hard to not have a confirmation bias when viewing polls, where you tend to accept a polling result that confirms what you believe or hope will happen and question a result that doesn’t fit with your map of the world. I have found the best way to do this is to first try to view the poll from the other side. Say you are a conservative. Start by thinking about how you would view the poll if you leaned left instead.
  • Next, always, and I mean ALWAYS, discover who paid for the poll. If it is an entity that has a vested interest in the results, such as a campaign, a PAC, and industry group or lobbyist, go no further. Don’t even look at the poll. In fact, if the sponsor of the poll isn’t clearly identified, move on and spend your time elsewhere. Good polls always disclose who paid for it.
  • Don’t just look to who released the poll, review which organization executed it. For the most part, polls executed by major polling organizations (Gallup, Harris, ORC, Pew, etc.) will be worth reviewing as will polls done by colleges with polling centers (Marist, Quinnipiac, Sienna, etc.). But there are some excellent polling firms out there you likely have never heard of. When in doubt, remember that Five Thirty Eight gives pollsters grades based on their past performances.  Despite what you may hear, polls done by major media organizations are sound. They have polling editors that understand all the nuances and have standards for how the polls are conducted. These organizations tend to partner with major polling organizations that likewise have the methodological muscle that is necessary.
  • Never, and I mean NEVER, trust a poll that comes from a campaign itself. At their best, campaigns will cherry pick results from well executed polls to make their candidate look better. At their worst, they will implement a biased poll intentionally. Why? Because much of the media, even established mainstream media, will cover these polls. (As an aside, if you are a researcher don’t trust the campaigns either. From my experience, you have about a 1 in 3 chance of being paid by a campaign for conducting their poll.)
  • Ignore any talk about the margin of error. The margin of error on a poll has become a meaningless statistic that is almost always misinterpreted by the media. A margin of error really only makes sense when a random or probability sample is being used. Without going into detail, there isn’t a single polling methodology in use today that can credibly claim to be using a probability sample. Regardless, being within the margin of error does not mean a race is too close to call anyway. It really just means it is too close to call with 95% certainty.
  • When reading stories on polls in the media, read beyond the headline. Remember, headlines are not written by reporters or pollsters. They are written by editors that in many ways have had their journalistic integrity questioned and have become “click hunters.” Their job is to get you to click on the story and not necessarily to accurately summarize the poll. Headlines are bound to be more sensational that the polling results merit.

All is not lost though. There are plenty of good polls out there worth looking at. Here is the routine I use when I have a few minutes and want to discover what the polls are saying.

  • First, I start at the Polling Report. This is an independent site that compiles credible polls. It has a long history. I remember reading it in the 90’s when it was a monthly mailed newsletter. I start here because it is nothing more than raw poll results with no spin whatsoever. Their Twitter feed shows the most recently submitted polls.
  • I sometimes will also look at Real Clear Politics. They also curate polls, but they also provide analysis. I tend to just stay on their poll page and ignore the analysis.
  • FiveThirtyEight doesn’t provide polling results in great detail, but usually draws longitudinal graphs on the probability of each candidate winning the nomination and the election. Their predictions have valid science behind them and the site is non-partisan. This is usually the first site I look at to discover how others are viewing the polls.
  • For fun, I take a peek at BetFair which is an UK online betting site that allows wagers on elections. It takes a little training to understand what the current prices mean, but in essence this site tells you which candidates people are putting their actual money on. Prediction markets fascinate me; using this site to predict who might win is fun and geeky.
  • I will often check out Pew’s politics site. Pew tends to poll more on issues than “horse race” matchups on who is winning. Pew is perhaps the most highly respected source within the research field.
  • Finally, I go to the media. I tend to start with major media sites that seem to be somewhat neutral (the BBC, NPR, USA TODAY). After reviewing these sites, I then look at Fox News and MSNBC’s website because it is interesting to see how their biases cause them to say very different things about the same polls. I stay away from the cable channels (CNN, Fox, MSNBC) just because I can’t stand hearing boomers argue back and forth for hours on end.

This is, admittedly, way harder than it used to be. We used to just be able to let Peter Jennings or Walter Cronkite tell us what the polls said. Now, there is so much out there that to truly get an objective handle on what is going on takes serious work. I truly think that if you can become an intelligent, unbiased consumer of polls it will make you a better market researcher. Reading polls objectively takes a skill that applies well to data analysis and insight generation, which is what market research is all about.

Why Lori is the Best Shark in the Tank

Shark Tank is one of my favorite TV shows. It showcases aspiring entrepreneurs as they make business presentations to an investor panel, who then choose whether to invest. It is fun to play along and try to predict how the Sharks will react to a business pitch. As a small business owner/entrepreneur, it is fun to imagine how I might do in a Shark Tank presentation. And, it was interesting to watch a college teammate of mine make a successful pitch on the show.

My inner geek came out when watching a recent episode. I got to wondering how much the need to entertain might cloud how the venture capital world is portrayed on the show. How many Shark Tank pitches actually result in successful companies? Is the success rate for Shark Tank businesses any higher than any other small company looking for growth capital? Are there any biases in the way the Shark Tanks choose to invest?

This curiosity led to a wasted work day.

Venture capital, especially at early stages, involves high risk bets. Firms may invest in 100 companies knowing full well that 80 or 90 of them will fail, but that a handful of wild successes will pay off handsomely. It isn’t for the faint of heart. I found an interview with Mark Cuban where he stated he hoped that 15% of his Shark Tank investments would eventually pay off. Even that seems high. Given that he has invested about $32 million so far, that is an admission that $27.5 million of that is expected to be wasted. Gutsy.

I also was able to discover interesting things about the show that are largely hidden from the viewer:

  1. The Sharks themselves are paid to take part. I was able to find discussions that suggested they may make as much as $100K per episode. That is a million dollars or more per season, so perhaps they are playing with house money more than they let on.
  2. Getting on Shark Tank is statistically harder than getting into an Ivy League college. It is estimated that more than 50,000 people apply for each season with less than 1% being successful. That alone should provide some realism as to the probability of success of new businesses.
  3. In the early seasons, an entrepreneur had to give up 2% of revenue or 5% of his/her company to the production firm just to appear on the show. That requirement was removed in later seasons because Mark Cuban refused to remain on the show if it remained.
  4. Many of the deals you see made on the show don’t end up being consummated. Forbes conducted survey research in 2016 that indicated that 43% of Shark Tank deals fell apart in the due diligence stage and 30% of the time the deal changed substantially from what is seen on TV. The deal you see on TV only came to fruition as you saw it about 1 in 4 times (27% of the time).

This makes it challenging to assess the deals and whether or not they paid off. Shark Tank companies are almost all privately-held so their revenue data is tough to come by and we can’t really know for sure what the deal was.

Although we can’t review business outcomes as we might like, we can look closely at the deals themselves. The data we used for this includes all deals and prospective deals from the first nine seasons of the show. So, it does not include the current season, which premiered in October 2018.

In the first nine seasons, there were 803 pitches resulting in 436 closed on-air deals (53% of pitches). Applying the Forbes data would imply that of these 436 deals, 187 of them likely feel apart, and 131 of them likely changed substantially. The net? Our projection would be that 53% of pitches result in handshakes on-air, but post-show only 37% of all original pitches close at all and only 15% of pitches will close at the terms you see on air.

Why would Shark Tank deals fail to close? There is a due diligence stage where investors get to have their accountants review the entrepreneur’s books. I found some articles that indicated that some entrepreneurs got cold feet and refused the deal after the show. Also, some of the deals have contingencies which fail to occur.

It is interesting to look at deals by the gender of the entrepreneur as it shows that Shark Tank entrepreneurs skew heavily male:

  • Men are much more likely than women to appear as entrepreneurs on Shark Tank. Of the 803 pitches, 482 (60%) made by men, 198 (25%) by women, and 119 (15%) by mixed teams of men and women. So, 75% of the time, at least one male is involved in the pitch, and 40% of the time at least one female is involved in the pitch.
  • However, women (59% closed) are more likely than men (51%) to successfully close a deal on air.

There are data that imply that men and women negotiate differently:

  • Men initially ask for higher company valuations ($4.5 million on average) than women ($3.1 million on average).
  • Men also ask for more capital ($342K on average) than women ($238K on average).
  • Men (47%) and women (49%) receive about the same proportion of their initial valuation ask. Men (94%) and women (88%) also receive about the same proportion of cash that they initially ask for.

So, men are far more likely to appear on the show and come with bigger deals on average than women. But they receive (proportionately) about the same discount on their deals as women as they negotiate with the Sharks. If there is a difference in their negotiation skills it is that men start bigger or come to the show with more mature companies.

We can also look at individual Sharks to get a sense of how good of negotiators they are:

  • Mark is the most aggressive Shark. He has the most deals (132, despite not being on the early seasons of the show) as well as the most invested (about $32 million).
  • The cheapest (or most frugal?) Sharks are Barbara and Daymond. Barbara has put forth the least amount of money (about $10 million) and her average deal valuation is $945K. Daymond has put out the second least amount of money (about $12 million) and has an average deal size of $957K. These two Sharks have likely not put much more money into their Shark investments than they have been paid to be on the show.
  • Mr. Wonderful seems to have a “go big or stay home” mentality. He has closed the fewest deals (64) of any Shark. But, his average deal valuation of $2.7 million is the highest of any Shark.
  • Lori and Kevin (31% of pitches) are the most likely to make an offer. Barbara and Daymond (22%) are least likely to make an offer.
  • So, Kevin make the most offers and closes the fewest deals, making him the least desirable Shark from the standpoint of the entrepreneurs.

Barbara is the most likely to invest in a female entrepreneur. She is about as likely to invest in a female entrepreneur as a male entrepreneur despite the fact that so many more men than women appear on the show. Kevin and Robert are the least likely to invest in a female entrepreneur. Mark and Daymond demonstrate no bias, as the invest in about the same proportion as appearances on the show.

ALL

Barbara Lori Mark Daymond Kevin

Robert

Male

60%

44% 53% 60% 57% 67%

71%

Female

25%

42% 33% 27% 25% 19%

17%

Mixed Team

15%

14% 14% 13% 18% 14%

12%

So, who has been the most successful Shark? It can be hard to tell because data are scarce, but my vote would go for Lori. USA Today put out a list of the top 20 best-selling products featured on Shark Tank. Six of the top 10 were from investments Lori made, including the top 3. Eight of the top 10 investments by revenue were made by the two female Sharks, Lori and Barbara.

Who are the worst Sharks in terms of revenue performance? My vote here would be a tie between Mark and Daymond. Mark has just 3 of the top 20 investments and Daymond has just 2. If we can assume that the goal of venture capital is to generate big wins, it is clear that Lori and Barbara are killing it and Mark and Daymond are not.

Shark Tank is a great catalyst for entrepreneurs, but because it is entertainment and not reality it can mischaracterize entrepreneurship in the real world. Sharks may invest for the entertainment value of the show and because investing boosts their personal brand as much as the product. And, it might just be the case that the amount of money they have invested is not much larger than the amount of money they have been paid to be on the show.

Almost all successful people will tell you that learning from their failures was at least as important as their successes, yet Shark Tank never revisits failed investments and it is likely that the bulk of the deals we see on TV do not end up paying off for the investor. The show does not disclose how few of its deals actually come to fruition once the cameras are no longer rolling. Just once I’d like to see an update segment show an investment that failed miserably.

Shark Tank also seems to imply that hard work and grit always triumph, when in reality knowing when to cut losses and having a little bit of luck matters a lot in business success. Grit matters for sure, but not when it’s focus is blind and irrational, and it can be sad to see entrepreneurs who have sacrificed so much and it is clear their business is not going to make it.

At its best, Shark Tank stimulates people to think like an entrepreneur. At its worst, it presents too rosy a picture of small business life which influences people to invent new products and launch companies that are likely to fail, at great consequence to the entrepreneur. It certainly provides great entertainment.

Will Blockchain Disrupt Marketing and Research?

The field of survey research was largely established in the 1930s and matured in the post WWII era as the US economy boomed and companies became more customer-driven. Many early polls were conducted in the most old-fashioned way possible: by going door-to-door with a clipboard and pestering people with questions. Adoption of the telephone in the US (which happened slowly – telephone penetration was less than 50% before WWII and didn’t hit 90% until 1972) made possible an efficient way to reliably gather projectable samples of consumers and the research industry grew quickly.

Then the Internet changed everything. I was fortunate to be at a firm that was leading the charge for online market research at a time when Internet penetration in the US was only about 20%. By the time I left the firm, Internet penetration had reached over 85% and online market research had pretty much supplanted telephone research. What had taken the telephone 40+ years to do to door-to-door polling had happened in less than 10 years, completely transforming an industry.

So, what is next? What nascent technology might transform the market research industry?

Keep your eyes on Blockchain.

Blockchain is best known as the technology that underpins cryptocurrencies like Bitcoin. The actual technology of Blockchain is complex and difficult for most people to understand. (I’d be lying if I said I understood the technology.) But, Blockchain is conceptually simple. It is a way to exchange value and trust between strangers in an un-hackable way and without the need for middlemen. It allows value to be exchanged and stored securely and privately. Whereas the Internet moves information, Blockchain moves value.

Those interested in the potential for Blockchain technology should read The Blockchain Revolution by Don and Alex Tapscott. Or, if you’d like a shortcut, you can watch Don’s excellent Ted Talk.

If Blockchain gains steam and hits a critical mass of acceptance, it has the potential to transform everything including our financial system, our contracts, our elections, our corporate structures, and our governments. It has applicability for any aspect of life that involves an exchange of value that requires an element of trust – which is pretty much everything we do to interact as human beings.

A simple example of how it works is provided by its first widespread application – as a cryptocurrency like Bitcoin. Currently, if I buy a book online, my transaction passes though many intermediaries that are often transparent to me. My money might move from my credit card company to my bank, to another bank, to Amazon, their bank, to the bookseller, to their bank, and I suppose eventually a few crumbs make their way to the author (via their bank of course). There are markups all along the way that are taken by all the intermediaries who don’t add value beyond facilitating the transaction. And, at every step there is an opportunity for my data to be compromised and hacked. The digital shadow left allows all sorts of third parties to know what I am reading and even where I am reading it.

This is an imperfect system at best and one that a cryptocurrency resolves. Via Bitcoin, I can buy a book directly from an author, securely, with no opportunity for others to see what I am doing or to skim value along the way. In fact, the author and I remain strangers.

Blockchain is mostly known currently as Bitcoin’s technology, but its potential dwarfs its current use. Blockchain will clearly transform the financial services industry, and for the better. Buyers and sellers can transact anonymously and securely without a need for intermediaries. Stocks can be exchanged directly by buyers and sellers, and this could lead to a world devoid of investment banks, brokers, and hedge fund managers, or at least one where their roles become strictly advisory.

A useful way to think of the potential of Blockchain is to think of trust. Trust in an economic sense lowers transactions costs and decreases risk. Why do I need lawyers and a contract if I can fully trust my contractor to do what he/she promises? Why do I need Uber when I can contract directly with the driver? As transactions costs decline, we’ll see a much more “democratized” economy. Smaller entities will no longer be at a disadvantage. The costs of coordinating just about anything will decline, resulting in a smaller and very different role for management. If Blockchain really ignites, I’d expect to see flatter corporate structures, very little middle management, and a greater need for truly inspirational leaders.

Any industry reliant on payment systems or risk is ripe for disruption via Blockchain technology. Retail, insurance, government contracting, etc. will all be affected. But, Blockchain isn’t just about payments.  Payments are just a tangible manifestation of what Blockchain really facilitates – which is an exchange of value. Value isn’t always monetary.

Which brings me (finally!) to our field: marketing and marketing research. Marketers and market researchers are “middlemen” – and any middleman has the potential to be affected by Blockchain technology. We stand between the corporation and its customers.

Marketers should realize Blockchain may have important implications to the brand. A brand is essentially a manifestation of trust. In the current digital world, many marketers struggle to retain control of their brands. This is upsetting to those of us trained in historical brand management. Blockchain will result in a greater focus on the brand by customers. They will seek to trust the brand more because Blockchain can enable this trust.

As a researcher I see Blockchain as making it essential that I add value to the process as opposed to being a conduit for the exchange of value. Put more simply, Blockchain will make it even more important that researchers add insight rather than merely gather data. In custom research about half of the cost of a market research project is wrapped up in data collection and that is the part that seems most ripe for disruption. There won’t be as many financial rewards for researchers for the operational aspects of projects. But, there will always be a need to help marketers make sense of the world.

When we design a survey, we are seeking information from a respondent. This information might be classification information (information about who you are), behavioral information (information about what you do), or attitudinal information (information about what you think and feel). In all cases, as a researcher, I am trusting that respondents will provide this information to me willingly and accurately.  As a respondent, you trust me to keep your identity confidential and to provide you an honorarium or incentive for your time. We are exchanging value – you are providing me with information and your time, and I am providing you with compensation and a comfort that you are helping clients better understand the needs of their customers. Blockchain has the potential to make this process more efficient and beneficial to the respondent. And that is important – our industry is suffering from a severe respondent trust problem right now. We don’t have to look much past our plummeting response rates to see that we have lost the respondent trust. Blockchain may be one way we can earn it back.

Blockchain can also authenticate the information we analyze. It can sort out fake data, such as fake postings on websites. To its core, Blockchain makes data transfers simple, secure, and efficient. It can help us more securely store personal information, which in turn will assure our respondents that they can trust us.

Blockchain can provide individuals with greater control over their “digital beings.” Currently, as we go about our lives (smartphone in pocket) we leave digital traces everywhere. This flotsam of our digital lives has value and is gathered and used by companies and governments, and has spawned new research techniques to mine value from this passive data stream. The burgeoning field of Big Data analysis is dependent on this trail we leave. Privacy concerns aside, it doesn’t seem right that consumers are creating a value they do not get to benefit from. Blockchain technology has the potential to allow individuals to retain control and to benefit from the trail of value they are leaving behind as they negotiate a digital world.

Of course as a research supplier I can also see Blockchain as a threat, as suppliers are middlemen between clients and their customers. Blockchain has the potential to replace, or at least enhance, any third-party relationship.  But, I envision Blockchain as being beneficial to smaller suppliers like Crux Research. Blockchain will require suppliers to be more value-added consultants, and less about reliable data collection. That is precisely what smaller suppliers do better than the larger firms, so I would predict that more smaller firms will be started as a result.

Blockchain is clearly in its infancy for marketers. Its potential may prove to be greater than its reality. But, just as we saw with the rise of the Internet, a technology such as this can grow up quickly, and can transform our industry.

Happy Birthday to Us!

happy-birthday-images-free-animated-free-animated-funny-happy-birthday-clip-arts-animated-butterfly-clipart-neoclipartcom-high-quality-pictures-dxeqzw

This month, Crux Research turns 11 years old. What started as something transitional for us as we looked for the next big thing quickly morphed into the next big thing itself.

Since our start, we have now conducted 300+ projects for 65+ clients across a wide range of industries and causes. At times, we feel we know a little bit about everything at this point.

We’ve bucked a few trends along the way. We’ve never had a business plan and have never really looked past the next few months. We’ve resisted pressure to grow to a larger company. We don’t necessarily go where the opportunities are and instead prefer to work on projects and with clients that interest us. We’ve also eschewed the normal business week, and work nights, weekends, etc.

Our ability to collect incredible people as clients has only been surpassed by our good fortune to attract staff and helpers. A special thanks to our staff members and our “bench” who have been helping out our team throughout the years.

Onward!  Happy Holidays to all. May your response rates be high and all of your confidence intervals be +/-5%!

An Epic Fail: How Can Pollsters Get It So Wrong?

picture1

Perhaps the only bigger loser than Hillary Clinton in yesterday’s election was the polling industry itself. Those of us who conduct surveys for a living should be asking if we can’t even get something as simple as a Presidential election right, why should our clients have confidence in any data we provide?

First, a recap of how poorly the polls and pundits performed:

  • FiveThirtyEight’s model had Clinton’s likelihood of winning at 72%.
  • Betfair (a prediction market) had Clinton trading at an 83% chance of winning.
  • A quick scan of Real Clear Politics on Monday night showed 25 final national polls. 22 of these 25 polls had Clinton as the winner, and the most reputable ones almost all had her winning the popular vote by 3 to 5 points. (It should be noted that Clinton seems likely to win the popular vote.)

There will be claims that FiveThirtyEight “didn’t say her chances were 100%” or that Betfair had Trump with a “17% chance of winning.” Their predictions were never to be construed to be certain.  No prediction is ever 100% certain, but this is a case where almost all forecasters got it wrong.  That is pretty close to the definition of a bias – something systematic that affected all predictions must have happened.

The polls will claim that the outcome was in the margin of error. But, to claim a “margin of error” defense is statistically suspect, as margins of error only apply to random or probability samples and none of these polls can claim to have a random sample. FiveThirtyEight also had Clinton with 302 electoral votes, way beyond any reasonable error rate.

Regardless, the end result is going to end up barely within the margin of error of most of these polls erroneously use anyway. That is not a free pass for the pollsters at all. All it means is rather than their estimate being accurate 95% of the time, it was predicted to be accurate a little bit less:  between 80% and 90% of the time for most of these polls by my calculations.

Lightning can strike for sure. But this is a case of it hitting the same tree numerous times.

So, what happened? I am sure this will be the subject of many post mortems by the media and conferences from the research industry itself, but let me provide an initial perspective.

First, it seems that it had anything to do with the questions themselves. In reality, most pollsters use very similar questions to gather voter preferences and many of these questions have been in use for a long time.  Asking whom you will vote for is pretty simple. The question itself seems to be an unlikely culprit.

I think the mistakes the pollster’s made come down to some fairly basic things.

  1. Non-response bias. This has to be a major reason why the polls were wrong. In short, non-response bias means that the sample of people who took the time to answer the poll did not adequately represent the people who actually voted.  Clearly this must have occurred. There are many reasons this could happen.  Poor response rates is likely a key one, but poor selection of sampling frames, researchers getting too aggressive with weighting and balancing, and simply not being able to reach some key types of voters well all play into it.
  2. Social desirability bias. This tends to be more present in telephone and in-person polls that involve an interviewer but it happens in online polls as well. This is when the respondent tells you what you want to hear or what he or she thinks is socially acceptable. A good example of this is if you conduct a telephone poll and an online poll at the same time, more people will say they believe in God in the telephone poll.  People tend to answer how they think they are supposed to, especially when responding to an interviewer.   In this case, let’s take the response bias away.  Suppose pollsters reached every single voter who actually showed up in a poll. If we presume “Trump” was a socially unacceptable answer in the poll, he would do better in the actual election than in the poll.  There is evidence this could have happened, as polls with live interviewers had a wider Clinton to Trump gap than those that were self-administered.
  3. Third parties. It looks like Gary Johnson’s support is going to end up being about half of what the pollster’s predicted.  If this erosion benefited Trump, it could very well have made a difference. Those that switched their vote from Johnson in the last few weeks might have been more likely to switch to Trump than Clinton.
  4. Herding. This season had more polls than ever before and they often had widely divergent results.  But, if you look closely you will see that as the election neared, polling results started to converge.  The reason could be that if a pollster had a poll that looked like an outlier, they probably took a closer look at it, toyed with how the sample was weighted, or decided to bury the poll altogether.  It is possible that there were some accurate polls out there that declared a Trump victory, but the pollster’s didn’t release them.

I’d also submit that the reasons for the polling failure are likely not completely specific to the US and this election. We can’t forget that pollsters also missed the recent Brexit vote, the Mexican Presidency, and David Cameron’s original election in the UK.

So, what should the pollsters do? Well, they owe it to the industry to convene, share data, and attempt to figure it out. That will certainly be done via the trade organizations pollsters belong to, but I have been to a few of these events and they devolve pretty quickly into posturing, defensiveness, and salesmanship. Academics will take a look, but they move so slowly that the implications they draw will likely be outdated by the time they are published.  This doesn’t seem to be an industry that is poised to fix itself.

At minimum, I’d like to see the polling organizations re-contact all respondents from their final polls. That would shed a lot of light on any issues relating to social desirability or other subtle biases.

This is not the first time pollsters have gotten it wrong. President Hillary Clinton will be remembered in history along with President Thomas Dewey and President Alf Landon.  But, this time seems different.  There is so much information out there that seeing the signal to the noise is just plain difficult – and there are lessons in that for Big Data analyses and research departments everywhere.

We are left with an election result that half the country is ecstatic about and half is worried about.  However, everyone in the research industry should be deeply concerned. I am hopeful that this will cause more market research clients to ask questions about data quality, potential errors and biases, and that they will value quality more. Those conversations will go a long way to putting a great industry back on the right path.

Cause Change. Be Changed.

Congratulations to Causewave Community Partners on their successful annual celebration last week.  It was a sellout!

This video does a great job of capturing what the organization is a about and the value of volunteering.  It also includes a cameo from Lisa on our staff!

 


Visit the Crux Research Website www.cruxresearch.com

Enter your email address to follow this blog and receive notifications of new posts by email.