Statistical vs Practical Significance Compared

What's the difference between Statistical versus Practical Significance? If you’re around data at all, you’ve likely heard someone talk about statistical significance. This hurdle is often considered the gold standard in a variety of disciplines for showing that something is true. A new drug being developed needs to show statistically significant efficacy to be considered viable. A new policy intervention is proved effective if it can, with statistical significance, promote a particular behavior. And a choice between different advertisements is made when one statistically significantly outperforms the other on some key metric like favorability.
But even though statistical significance is a minimum requirement for any of those conclusions, it fails to consider something else, meaningful significance. In other words, just because something is true, doesn’t mean it matters all that much. In this episode, if you stick around to the end, I’m going to try and provide an intuitive framework for how to think about whether a result of some kind isn’t just true, but also whether it’s meaningful.
Video explaining statistical significance: • Statistical Significan...
Video explaining randomized experiments: • Correlation Doesn't Me...
Learn more about who I am and why I'm doing this here: • Data Demystified - Who...
Video that has more detail wealth inequality: • The Data Behind Wealth...
Video that explains how to adjust for inflation: • Inflation Explained | ...
Follow me at:
LinkedIn: / jeff-galak-768a193a
Patreon: / datademystified
Equipment Used for Filming:
Nikon D7100: amzn.to/320N1FZ
Softlight: amzn.to/2ZaXz3o
Yeti Microphone: amzn.to/2ZTXznB
iPad for Teleprompter: amzn.to/2ZSUkNh
Camtasia for Video Editing: amzn.to/2ZRPeAV

[@OEV16Regional]

New subscriber here! I am in a Statistics course for Nursing for Evidence-Based Practice, and you are life-saver. Your explanation of these difficult to grasp concepts are very effective in helping me gain an understanding of some of the concepts. I look forward to browsing through your playlist to see if you have covered a concept in my course that my book does not do well in teaching.

[@santiagodm3483]

Thanks for the explanation, you did it quite well. I'm a Colombian student of economics, and I've learned a lot of statistics with this channel. It's interesting how in economics people tend to think that expensive policies despite being expensive and yields few results should remain, practical significance matters a lot.

[@merrytricious]

This is such a helpful video. I subscribed. I am so happy I found your channel to help me this semester.

[@ChiTwnCutie]

You just earned yourself a subscriber!!! Thank you so much for a superb video! The examples were excellent! I will be taking quantitative research next semester and I am confident your videos will be useful! Again, thank you!

[@DataDemystified]

So happy you found them useful! Happy holidays!

[@graceyoder1205]

This was so interesting! Thank You for putting in the effort to make this content - it has honestly piqued my interest!

[@DataDemystified]

Thank you. Glad you found it interesting!!

[@khyatichouksey8927]

Very well explained, I finally got it after 1.5 years!

[@DataDemystified]

Happy to hear that!

[@nhunghoang9872]

so touching for an excellent video

[@yohangarcia01]

Thank you!

[@himanshutyagi1107]

Hi Prof , Thank you so much for posting this great video. Could you please make a video on effect size as well ? Much Thanks.

[@DataDemystified]

Thanks! Not exactly what you're asking for, but in the ballpark; kzbin.info/www/bejne/b2awqneMrbylhZo

[@himanshutyagi1107]

@@DataDemystified Thank you:)

[@pinkisesmundo9275]

Thank you!!!! It finally is clicking. Whoa!!

[@AllBusiness37]

So statistical significance is a quantitative question or, in other words, a numbers question. Numbers, formulas, math, etc. And practical significance is a values question. If we determine that the action being studied has statistical significance (e.g., we determine that a vaccine actually reduces disease incidence and that the lower case numbers aren’t simply a product of chance), then we need to ask ourselves if that action is something that we, as a community, want to implement. The latter question, i.e., the practical-significance question, is about who we are, who we want to be, and how we want to allocate our limited resources. We might find it odd or unreasonable, but we can imagine a community that decides to dedicate enormous resources to lowering the incidence of a certain disease by only one percent. Different communities can answer the practical-significance question differently even though, in contrast, and because it is a quantitative question, they must answer the statistical-significance question in the same way. Is this a fair statement, or have I wandered off the path?

[@matthewbartsh9167]

I agree with everything you said. I would go further, perhaps, by saying that more or less everything in the human world is correlated with everything else, and therefore it only takes a big enough sample size to have a fifty (50) percent chance of showing statistical significance. Also, the interesting results are the ones where a large sample did *not* show a statistically significant correlation in either direction. This is because such a result shows an exceptionally small effect. For example, if there is no detected correlation, with a large sample, between life expectancy and eating a certain food, we might conclude it is safe to eat it, and reasonably. On the other hand if there is a statistically significant correlation with lowered life expectancy, that on its own (without knowing the effect size) tells us essentially nothing, or if it does tell us anything, it is only that the food does not increase life expectancy if the sample is large, which is the same as what finding no correlation either way tells us. Does any of that make sense?

[@DataDemystified]

I think that's a very clear understanding of statistical vs. practical importance! Thank you for the comment!

[@matthewbartsh9167]

@@DataDemystified If you have time, please read and comment on my recent Medium article "What Statistical Significance Really Means" that got accepted by Nerd For Tech magazine. If you want to give me feedback privately, DM me on Twitter. I am @bartshmatthew on the latter.