The New Statistics: Confidence Intervals, NHST, and p Values (Workshop Part 1)

[@JCResDoc94]

13:40 Results Approaching The Train Station To Get To The Airport To Get As Far Away From Significance As Fast As Possible...Or "Approaching" Significance - Ask Yourself?

[@WikiofScience]

Although I agree with Cumming's call for CIs and meta-analysis, I disagree with some of the assumptions in this video. I commented on that in a recent article in Frontiers in Psychology, and here goes some excerpts from that comment: Firstly, Cumming’s "dance of p’s ...is not suitable for representing Fisher’s ad hoc approach (and, by extension, most NHST projects). It is, however, adequate for representing Neyman-Pearson’s repeated sampling approach". The role of the p-value for each approach is different, for Neyman-Pearson's approach being "a simple count of significant tests irrespective of their p-value". Secondly, as it turns out, Cumming’s simulation is "a textbook example of what to expect given power", under Neyman-Pearson's approach). For example, 52% of tests should be significant at α = .05 in the long run, when power is set to 52%. Thirdly, Cumming doesn't compare p's and CI's fairly. "To be fair, a comparison of both requires equal ground. At interval level, CIs compare with power". While Cumming’s simulation reveals that about 95% of sample statistics fall within the population CI (out of 95% expected), 52% of those sample statistics are statistically significant (out of 52% expected). Furthermore, "at point estimate level, means (or a CI bound) compare with p-values, and Cumming’s figure reveals a well-choreographed dance between those. Namely, CIs are not superior to Neyman-Pearson’s tests when properly compared although, as Cumming discussed, CIs are certainly more informative." Fourthly, what is the truth the p-value provides? And, do CIs tell a better truth? "It is not clear what truth Cumming refers to, most probably about two known fallacies: that p is the probability of the main hypothesis being true and, consequently, that 1 - p is the probability of the alternative hypothesis being true (e.g., Kline, 2004). Similar fallacies equally extend to the power of the alternative hypothesis. Yet accepting the alternative hypothesis does not mean a 1 - β (a.k.a., power) probability of it being true, but a probability of capturing 1 - β samples pertaining to its population in the long run. The same can be said about CIs (insofar CI = 1 - β): they tell something about the data-about their probability of capturing a population parameter in the long run-not about the population-i.e., the observed CI may not actually capture the true parameter. Another fallacy touched upon is that p informs about replication. P-values only inform about the ad hoc probability of data under the tested hypothesis, thus '(they) have little to do with replication in the usual scientific sense' (Kline, 2004, p.66). Similarly, CIs do not inform about replicability either. They are a statement of expected frequency under repeated sampling from the same population." ---- Perezgonzalez JD (2015). Frontiers in Psychology (doi: 10.3389/fpsyg.2015.00034, journal.frontiersin.org/Journal/10.3389/fpsyg.2015.00034/full).

[@JCResDoc94]

WikiofScience JKAI Yer, but you would say that.

[@rocketrider1405]

I thought a p-value gives evidence in favour of the null hypothesis; and if the p-value is so low (say lower than your alpha level of 0.05) then you reject the null.

[@userou-ig1ze]

and the speaker says something else?

[@illywhacka]

Bayesian.