MikeOnBlog

If there is one thing that just burns me is when someone says with conviction that a number is statistically significant when none of the formal statistical evidence gathering has been done.  So… A primer on Statistical Significance (from someone whom isn’t that statistically bent), it’s importance, why being Statistically Significant isn’t always significant and why I nearly go into a blind rage when some one uses the term out of context.

I like wikipedia’s definition so I quote it here:

‘In statistics, a result is called significant if it is unlikely to have occurred by chance. “A statistically significant difference” simply means there is statistical evidence that there is a difference; it does not mean the difference is necessarily large, important or significant in the usual sense of the word.’

Or, in layman’s terms, something is statistically significant if what ever you did (or whatever external event might have happened) made a difference (this is where testing, usually A-B testing, comes into play) and it probably (I intentionally use the word “probably” as this is where the term “confidence interval” comes into play) wasn’t just blind luck. 

Why is knowing whether something is statistically significant is important?  Simply put, when you need to know if an event or a trend is the result of random variation or not.

Without going into the mathematics, three important factors contribute to a popular method of statistical significance testing (in this case the t-test) when comparing two independent data sets. These are the mean, or average, the number of data points (for some reason called degrees of freedom to Statisticians) and the range of these data points, or standard deviation.

For example let’s take the average live expectancy of two sets of rats.  One set gets a bowl of Chereos every day and the other doesn’t.  Even if the average of one set is noticeably (note, not significantly) larger than the other, a large standard deviation in each set may indicate that the difference in the average is not significant. [ probably should put some sample data in here]

What if there were a million rats in each sample and a bowl of Chereos every morning increased their life expentancy by 2.4 seconds.  While possibly statistically significant … it really doesn’t make a difference.  This is when statistical significance doesn’t really mean real world significance.

So why does this just frost me when someone says that something is statistically significant without doing the proof.  Because humans are wired to not deal with randomness very well.  We are wired to try to find patterns in randomness that don’t exist.  It seems 1/2 the world believes that magic patterns emerge from their Ipod song shuffle (”Dude… what are the chances of Dylan’s ‘A hard rain is gonna fall’ be followed by the Grateful Dead doing ‘Here Comes the Rain’ followed by CCR doing ‘Who’ll Stop the Rain’.  There is NO WAY that is random!  I’ve got 1000 songs on my Ipod…. blah.. blah.. blah..”) . Probably a survival trait we developed along our evolutionary path but this bias or tendency that, at best, doesn’t translate well into today’s reality and, at worst, makes for disastrous decision making.

OK… I’ve been writing this off and on for a week.  There is tons of stuff I didn’t touch on (hypothesis testing, Type 1 errors, Type 2 errors, p-values, etc) and I’m sure there are people much more familiar than me with these concepts.  If anyone ever read this blog, I’m sure I’d get some, hopefully, constructive feedback. Maybe I’ll follow this up with more details in the future.

Blogged with Flock