First of all, I loved the dance. Both my partner and I laughed at the random (or not so random) honking paired with emotionally labile dancers. A wonderful program. (sorry in advance, neither of us are statisticians, just trying to learn enough to do our job well) So my partner and I have a bit of disagreement resulting from watching this video. We were wondering about in what applications p-value shouldn't be trusted. Effect size & CI are a given as extremely important & required context. Assuming Bias to be negligible & sample size to be a sufficient for the field of research due to a wonderfully constructed experiment... One argument was that a p-value is not a great instrument for planning a study, but is better for planning management. For example, if I did a study & achieved p = 0.05 showing a +'ve correlation between poking you in the right eye & improving your tax returns then anyone who comes in with poor tax returns I could poke them in the eye & feel 19/20 likelihood to improve their tax returns. However if I attempted to prove this association with further research I may have difficulties. The other interpretation was that p-value is a poor tool for both research planning & developing management. The poor predictability on where p-value may fall in any one experiment means that its utility, even once obtained to be 'significant', lends poor predictive value for both.

I’m very pleased to have stumbled across this video and channel!

❤❤❤

very interesting. Thanks!

Thank you for your work.

Best statistics video ever! 💞

This is totally backwards. You are looking at the probability of getting a good p value if an effect is real. We want the opposite. How much more likely an effect, given a p value? If you ran such a simulation, you'd find the p value correlates with the existence of an effect better than any other measure. It's random, because data is random. Sometimes a few patients given a good drug will drop dead anyway. Your p value will be high because it should be. It's not a convincing result, get more data.

Ryan Faulk sent me here

Something that should be stated: psychology isn't some particularly pathological field. When you say "typical of psychology" - most people by default trust tagged authorities, and so will look for ANYTHING to contain the critique. It shouldn't be like this, they should be able to go, "you know, I don't actually know if any other separated field IS better than psychology in terms of mean or median sample size and/or effect size, so I can't ipso facto declare this problem DOES NOT also apply to every other field." But because it is, it's important to pre-respond to that containment tactic that most people will engage in.

I wish I could double like this. Amazing demonstration and fantastic educator!

Are this book and soft paid?

Watching this in 2021 and wow what a great explanation

Great video and clearly explained. Many thanks for your help Geoff!

Hi Geoff, you mentioned the width of the CI is predictive in some way of future CI widths, but what value is that to a researcher? Won't the range of the CI bounce around as shown in your simulation, and doesn't that imply that the confidence interval itself will miss the true mean as it shifts along the axis of possible means some percentage of the time? perhaps as often as the pvalue shifts?

Brilliant video

Has anyone found a jupyter notebook for this? If not I may have a go at it and let me know if you are interested in having a look. Thanks!

You are the best ever... really...

I love how he explains this so clearly. Clears out my anxiety about stats! Thanks so much Geoff!

Dr. Cumming, this is a job well done. :)

You are the best instructor ever... Why did I just find this T_T Where were you... hahah

Thanks for this really clear and interesting video! :)

amazing!

That's pretty amazing Excel magic there.

Hey sir I’m in your class

Thx man. Those vids are good

It would have been important to also mention the fact that the number of observations drawn from the population (i.e. the sample size) increases the "predictability" os inferences from the p-value.

Alt-Hype brought me here.

Geoff, what you have done here is a very misguided analysis. You have simply shown that statistical testing based on inadequate sample sizes does not make sense. Question: did you purposefully chose sample sizes 32 leading to the power of only 0.52! Your last sentence in the video should have been: this is a classical case not to perform underpower studies, not that the p-values have erratic behavior. In addition, you have selected huge standard deviations (relative to the means), hence there is a substantial amount of overlap between two populations. Suggestion 1: Increase the sample sizes, achieve better power, and your p-value dance will be totally different. Suggestion 2: Decrease both standard deviations to 10 and keep the sample sizes unchanged (n=32). You will be highly disappointed when you see the results: no p-dances at all. Almost all p-values will be < 0.05. Your video about "p-dances" should be retitled to "how to intentionally perform bad statistics".

Where can I get that tshirt!?

10/10

Thank you Geoff. That was a nice summary

thanks for amazing videos sharing. I liked your simple and warm presentations

Is it possible to have the first sample mean direct on the bottom like in the version before?

How many of y'all teachers sent you here zzzzzzzzz

Thanks for this Geoff, I was wondering if you were reporting both CI and p-values what interpretation would you trust when they conflict, so if CI don't cross zero but the p value is greater than 0.05?

Such an eye opener! I'm so glad you came across all of this! The implications are just staggering!

Brilliant!

That is a sick freakin beat my dude

Thanks for this Geoff - do you have any spread sheets if you have a mixed design? Thanks

Hi, Does the first book add anything to the second?

Somebody at a party😂😂😂

Hi Geoff, is the estimation process affected by Type 1 or 2 errors when making multiple comparisons (i.e. in a crossover with more than a pre and post measurements)?

really really helpful

UR A FKN ANIMAL. thanks so much

Some questions: Up front you mentioned Estimation being good... but it is not mentioned again. Does this mean confidence intervals? And... your presentation seems to assume known statistics. How would you handle drug testing, where only the null hypothesis statistics are known (only random effects), because the positive hypothesis (it improves outcomes) is of unknown strength and cannot be modelled. That is why drug trials use only the null hypothesis as set p = .05 in advance.

Dear Prof Cummings, thank you so much for this video and you outstanding paper on . Understanding the new statistics. Effect sizes, confidence intervals, and meta-analysis" This was truly an astounding piece of work, so well written, even a simple clinician like myself could understand it Thanks Johnny

"quite large for PSYCHOLOGY"

Straw Man? That's not at all what p-values are supposed to mean. Those are errors of human interpretation. Stats profs TRY to teach students how to use NHST, but most are abysmal teachers and most students blot out unpleasant Stats class memories anyway. Therein lies the problem. Then researchers go on to p-hack or overly rely on p-values on their rushed journey to academic fame, not knowing what they're doing. P-value is not the strength of evidence. It's simply the probability of observing an effect that size due to sampling error given the null hypothesis is true, i.e. the chance of making a Type I error if you take that effect seriously. No one ever said it was a standalone beacon of definitive proof. Just a marker for Type I error. But wait, why number error? Oh yeah, there's a type II error too. Anyone remember that from Stats class? From most publications in psychology, social sciences and life sciences, you'd think not. Given there are 2 types of error in NHST, modern researchers' single-minded obsession with type I error seems incredibly naive. Type II error vastly increases as sample size shrinks and as you try to detect smaller "true" effect sizes. Both types contribute to whether one may be wrong in interpreting sample data. In your simulation, the P-value is not very replicable due to type II error. You're not measuring Type I error (since the null hypothesis is not true by design). Under your conditions, replicability of low p-value would be a measure of statistical power (the chance of getting a low p-value given there is a true effect; the complement of type II error). But with low sample (n=32), type II error can be very high and power can be very low. So there becomes a high chance of observing a "not significant" p-value upon replication, despite the true effect in the population. The test simply lacks power to consistently detect the effect. This is all you're showing with the "dance". Remember that p > 0.05 does not mean no effect. It just means your sample lacked the evidence to show it ("fail to reject H0" not "accept H0"). P-value is very consistent with definition if simulated properly (i.e. under null hypothesis). If the true effect size is zero, you'd see 5% or fewer p-values < 0.05 after a "large" number of replications. Exactly what one would expect. What your simulation has uncovered is simply the elephant in the room with NHST, the issue of statistical power that too many researchers ignore. A meta-analysis of recent behaviour ecology publications showed the median power was well under 40% (i.e. chance of Type II error > 60%!!). NHST is not at fault here. Its rules clearly indicate the result is garbage, if one chooses to remember them. Also note most classical statistics rely on asymptotic (large sample) properties. NHST invokes an assumption of Normality of the mean (or regression coefficient or whatever). While Central Limit Theorem proves that true asymptotically, it may be very far from Normal with small samples. With n=32 or lower, you not only get more type II errors but these large sample properties also start to break down and parametric model assumptions may fail. There is a much deeper flaw in experimental design if relying too much on large sample behaviour with small sample data. Though that seems common practice in Psychology for whatever reason. Also note type I and type II errors are just related to sampling errors. Then there's misspecification (applying the wrong statistical distribution or model to the data). And non-sampling errors also exist, often dwarfing sampling error. A dedicated attempt at error analysis, model diagnostics and search for latent biases/confounders should preclude any hypothesis testing. Otherwise garbage in, garbage out. NHST is fine if used properly. However, in using it most researchers appear to be more clueless than Alicia Silverstone. The method is not wrong, just blindly misused.

I like this guy. Often statistics is presented as so exacting... but it isn't, it shows trends, and levels of noise, and to some degree it's far more valuable to get some analysis of data than to worry about minutia indefinitely.

Wait wait wait, this is all fine and dandy. But what about if there is no effect size, how is the distribution of p-values then? I mean, this is only proves that a medium effect size with n = 32 cannot be found 48% of the time. Isn't the p-value supposed to be 'the final arbiter' in the idea that it is hard to get to a low p-value when there really is no effect? I make the following conventional bet: if the effect size is low (e.g. .1), then the chance of obtaining a p-value smaller than .05 will be around 10%.