Permutation Hypothesis Testing with Example | Statistics Tutorial # 37 | MarinStatsLectures

Permutation Hypothesis Testing in Statistics with Examples: What is a Permutation Test? How to Do a Permutation Test in Statistics and Research? 👉🏼Related: Bootstrapping Concept and Bootstrapping in R Videos: (bit.ly/2GL6AYS) ; Permutation Hypothesis Test with R • Permutation Hypothesis... 👍🏼Best Statistics & R Programming Language Tutorials: ( goo.gl/4vDQzT )
🎗 Like to support us? You can Donate (bit.ly/2CWxnP2), Share our Videos, Leave us a Comment, Give us a Like or Write us a Review! Either way We Thank You!
In this video, we present the general concept of a Permutation Test. Permutation tests also get referred to as “Exact Hypothesis Tests”, and serve as an alternative approach to large-sample parametric approaches.
In mathematics, a permutation is the act of taking the members of a set (in this case, the set of all observations from the sample), and rearranging or reordering them. The total number of permutations for a set is the total number of unique re-orderings of the elements of a set. For example, if a set consists of (A, B, C), there are 3!=6 total permutations (ABC, ACB, CAB, BAC, BCA, CBA).
A Permutation hypothesis test makes use of the concept of permutations. The underlying concept of a Permutation Test is as follows. First, a test statistic is defined, and calculated for the observed sample data. Then, all possible permutations of the data are considered, and the test statistic is calculated for all possible permutations of the data. A p value is then calculated as the total percentage of permutation test statistics that are greater than or equal to the test statistic calculated for the observed sample data. In this sense, the p value is helping us to get an idea of how extreme the observed test statistic for the sample data is.
Because a permutation test considers all possible permutations, it gets referred to as an exact test, and calculates an exact p-value. In a real world, the total number of permutations for a set of data is extremely large, and in practice a random sample of all permutations is generally taken to work with (so in this sense, a permutation test often results in an approximate p value, even though it is usually referred to as being an exact test).
In this statistics video tutorial, we lay out the general procedure and concept of a permutation test. While the idea is presented in reference to a specific example (with X=categorical/factor/qualitative, and Y=numeric/continuous/quantitative), we present the concepts as generally as possible, so that the concept can be easily extended to permutation tests using different types of variables.
Accompanying this video is a separate video implementing the permutation test using the statistical software R (using RStudio).
► ► Watch More:
► Bootstrapping & Resampling Concept and with R: bit.ly/2GL6AYS
► Intro to Statistics Course: bit.ly/2SQOxDH
►Data Science with R bit.ly/1A1Pixc
►Getting Started with R (Series 1): bit.ly/2PkTneg
►Graphs and Descriptive Statistics in R (Series 2): bit.ly/2PkTneg
►Probability distributions in R (Series 3): bit.ly/2AT3wpI
►Bivariate analysis in R (Series 4): bit.ly/2SXvcRi
►Linear Regression in R (Series 5): bit.ly/1iytAtm
►ANOVA Concept and with R bit.ly/2zBwjgL
►Hypothesis Testing: bit.ly/2Ff3J9e
►Linear Regression Concept and with R Lectures bit.ly/2z8fXg1
Follow MarinStatsLectures
Subscribe: goo.gl/4vDQzT
website: statslectures.com
Facebook:goo.gl/qYQavS
Twitter:goo.gl/393AQG
Instagram: goo.gl/fdPiDn
Our Team:
Content Creator: Mike Marin (B.Sc., MSc.) Senior Instructor at UBC.
Producer and Creative Manager: Ladan Hamadani (B.Sc., BA., MPH)
These videos are created by #marinstatslectures to support some courses at The University of British Columbia (UBC) (#IntroductoryStatistics and #RVideoTutorials for Health Science Research), although we make all videos available to the everyone everywhere for free.
Thanks for watching! Have fun and remember that statistics is almost as beautiful as a unicorn!

[@marinstatlectures]

In this video we present the general concept of a Permutation Test in Statistics. Permutation tests also get referred to as “Exact Hypothesis Tests”, and serve as an alternative approach to large-sample parametric approaches. Like to support us? You can Donate (bit.ly/2CWxnP2), Share our Videos, Leave us a Comment and Give us a Thumbs up! Either way We Thank You!

[@alinamarinca7658]

How to do permutation in R for a regression test

[@samuelamissah8918]

Wait..What..My lecturer taught this in class. I didn't get it. Miraculously you guys uploaded one I can comprehend..Thanks!!!!!!!!!!!!!!

[@marinstatlectures]

You’re welcome, glad we could help out :)

[@kpuc]

Amazing lecture! Great and clear examples! Thank you very much!

[@SSWang0131]

Very clear and helpful STEP by STEP video for learning statistical concepts. Thank you for making such high-quality video!

[@MelodyYing-q4f]

Thank you! It is very clear

[@syhusada1130]

Thank you. So the only thing differ from this to bootstrap is that permutation is without replacement and bootstrap is with replacement? Both are resampling from the joined pool of both meatmeal and casein since both resampling are done to project the distribution from the sample data if the Null Hypothesis is true?

[@Viclevicking]

Very well explained, thank you!

[@sfdv1147]

Can you explain when would we prefer this over the normal Bootstrap approach? Thank you in advance

[@darktemplar2827]

Marin, your videos are always top notch.

[@marinstatlectures]

Thanks!

[@KARINEMOOSE]

This was a better explanation than my grad-school level class on the subject. Thank you, SO much!!!

[@marinstatlectures]

I’ve created these for the grad courses I teach :)

[@KARINEMOOSE]

@@marinstatlectures They are excellent! Thank you so much for sharing the resource, it will go a long way for so many!

[@shaonlahiri7805]

Really clear video! Thanks!

[@stephenday4834]

So clear and easy to understand! Thank you!

[@JaretH00]

thank you! finally an easy to grasp explanation :)

[@PatrickBateman12420]

Brilliant. Thank you for sharing your knowledge with us!

[@daesoolee1083]

I wish you were my professor for the statistics lecture during my Bachelor course.

[@touqeerahmad7837]

Very helping video thank you so much

[@NadiaALOUI-y3j]

Thank for the video! I have a question please : can we apply the permutation test for dependent samples?

[@xinking2644]

thank you very much, it really helps a lot!

[@marinstatlectures]

You’re welcome

[@mirexius]

You're awesome :)

[@jaredkachlany]

thank you!

[@Brockdorf]

Can you do a permutation test with skewness?

[@vaibhavchhaya9145]

Nice video! I was just wondering how one can construct the confidence interval for a permutation test? If I have calculated the pseudo- F-statistic using PERMANOVA, is there a way I can generate confidence intervals for it?

[@froggomcfroggin9392]

he says in the video how you cannot calculate CIs

[@bozhou1454]

@@froggomcfroggin9392 any idea on why?

[@buihung3704]

@@bozhou1454 The Confidence Interval needs to preserve the relationship/dependency between the Weight and Diets, but in Permutation Test we nullify it, make them independent. Now i know you gonna ask if we did permutation per group, instead of the whole data, well that's just re-shuffling the original question, doesn't change anything :))))

[@maxpedersen1612]

Hi! Why do we use a one-tailed test and not a two-tailed test?

[@yedidyashimon7275]

Brilliant.

[@mugunthan4152]

Sir, what is the p value here I don't understand the final steps

[@AndrywMarques]

Why is not possible to calculate the confidence interval using this approach but it's possible using the bootstrap approach?

[@ZinzinsIA]

Interesting but how can TS1 have different values after permutation, regarding that Yc and Ym are always the same, unless when you do the permutation you consider the diet being different ?

[@marinstatlectures]

Because under H0 you are assuming that weight is not related to the feed type, and so you are assuming that observations from one group are as likely to be from the other group. So you permute all observations across all the groups. Imagine the data in the standard set up where you have one column for group (feed type) and another column for weight. You keep the groups/labels of feed type fixed, and then you look at all permutations of the weight column. You always have the same observed weights, but the group they end up in changes. Hope that makes sense.

[@ZinzinsIA]

@@marinstatlectures ok thank you very much !

[@팬더-n4w]

why does the difference in means need to be absolute value, considering the distributions of the mean difference should be normal ?

[@mugunthan4152]

If we got Ts 40 in our 3rd test what is the pnvalue

[@jiehu1337]

Can we build the confidence interval like what we did in bootstrap method.

[@fishycoffe]

Thank you for your comprehensible explanation! I'm a little confused by the number of permutations. I read that you should take all possible combinations, meaning in your example 9 over 4 = 126 permutations. What's the difference to 10^4 permutations? Why would one do that if it doesn't give you any more information than 126 permutations?

[@petraborovska7266]

Thanks for that question, I was also wondering about that. In the video total number of permutation of both groups together is used, which gives you individual ordering of all 9 observations, but then you have in the data set also some redundant numbers which don't improve the information. So make sense to use only the individual ordering - and as you mentioned that would be 126 permutations. So I probably missing some peace of information to make sense of it. Other sources seem to be a bit inconsistent and sometimes I see this approach - as in video and sometimes the one r.a.w. was suggesting. But perhaps that is always dependent on how we state the hypothesis and test statistics, and that's why it is individual to particular scenario. Also I don't quite understand why would we talk in the second case about permutations, when the formula is a formula for combination with no repetition = n!/r! x (n-r)!. I would appreciate if someone could correct me or provide some hint, as I'm surely missing something. But otherwise the video is great, thanks a lot!

[@andachamamci9967]

@@petraborovska7266 I have the same question ! Did you able to find an answer Petra ?

[@콘충이]

Thanks!!

[@AndrywMarques]

So, the main difference to the bootstrap method is that one uses sampling with replacement and the other doesn't, right?

[@ftedg]

I think so... But apparently, this will allow us to build up hypothesis based on "complex" test statistics. That's what I understood

[@alonsom.donayre1992]

Hi your videos are amazing would continue doing them?

[@marinstatlectures]

thanks! yes, we plan to keep on making videos. with school back on, things are a bit busy with the courses I'm teaching, but we do have a few things in the works and hope to be releasing more videos as soon as we can

[@martincao5322]

So no one is gonna talk about how he's writing the script backward ?

[@giulionenna]

speechless

[@ZwergDesign]

He doesn't. The video is flipped...

[@stephenday4834]

There is a transparent screen in front of him. He writes normally on it with something like a whiteboard marker. The video is flipped left to right so the text is reversed. You may notice his shirt buttons are on the wrong side and he appears left handed. Flipped video.

[@xinhuang5759]

can this P-value be called empirical p-value?

[@ayoubmarah4063]

how much the p-value should be equal so we can say its statisticly significant ??

[@marinstatlectures]

the general guideline is less than 5% is statistically significant....although it is good to not be so rigid in a decision with the p-value. id recommend searching and reading the "American Statistical Association Statement on P-values". it was written a few years back now, but has a great discussion on the use (and misuse) of p-values

[@ayoubmarah4063]

@@marinstatlectures thank you for your help i appreciate that

[@hanyferdinando112]

@@marinstatlectures I understood the same way, small p-val indicates its significance. From your tutorial, I knew how to calculate p-val and the value based on the example is 0. What would you say from this p-val related to the experiment in your example? In this case, p-val is 0 and statistically significant. Does it mean that these two diets give different results? To me, p-val = 0 in this example that there is no significant difference between M and C. I ran a script and choose permutation, P, up to 10000 and I got the p-val is around 0.3-0.5.

[@jameschrostowski4884]

I believe the number of permutations should be 9!/(4!)(5!) and not 9! for the difference.. Please reply..

[@enjoying-the-ride1295]

Thank you for the detailed explanation, really helpful! I just wish I can hear your voice better, at times it is difficult to hear the words clearly and I had my volume up at 100%

[@jameschrostowski4884]

Or maybe number of permutations should be 2^9? I dont think so Like 9!/4!5! Better