In all of our dealings with clients, there is one question that we wince at:  “is this study statistically significant?”

This question is cringeworthy because it is challenging to understand what is meant by the question and the answer is never a simple “yes” or “no.”

The term “significant” has a specific meaning in statistics which is not necessarily its meaning in everyday English. When we say something is significant in our daily lives, we tend to imply that it is meaningful, important, or worthy of attention. In a statistical context, the meaning of the term is narrow:  it just means that there is a high probability our findings are not due to chance.

For instance, suppose we conduct a study and find that 55% of women and 50% of men prefer Coke over Pepsi. Researchers will tend to say that there is a statistically significant difference in the Coke/Pepsi gender preference as long as there is a 95% probability or better than these two numbers are different.

I won’t bore you with the statistical calculation, but in this case we would have had to interview about 1,000 women and 1,000 men in order to highlight this difference as being significant.

But, just because these two findings are statistically significant, doesn’t necessarily imply that they are practically important.  Whether or not a 5 point difference between men and women is something worth noting is really a more qualitative issue. Is that a big enough difference to matter? All we can really say as researchers is that yep, odds are pretty good the two numbers are different.

And that is where the challenge lies. Statistical significance and practical importance are not necessarily the same thing. Statistical significance is calculated mainly by knowing the sample size and the variance of response.  The more people you interview and the more they tend to have the same answers, the easier it is to find statistically significant differences.

The custom is to only highlight differences with a 95% or greater probability of being not being due to chance. But this is nothing more than a tradition. There is no reason not to highlight differences with a greater or less probability.  In a sense, every study that is implemented well is provides statistically significant results – it just depends on how much of a chance you are willing to take of making an error.

I recently had a client ask me what it would take to have a 0% chance of being wrong. The short answer is you would have to interview everybody in the population. And do it in a perfect, no-biasing way.

So, the correct answer to “is this study significant” is “it depends on how certain you want to be.” That is rarely a satisfying response, which is why we don’t like the question to begin with!