Hypothesis testing Practical Implementation|Hypothesis testing with data example in python

[@kshitijdesai2402]

This is the best video for practical hypothesis testing that I have watched!

[@UnfoldDataScience]

Thanks Kshitij

[@isaackodera9441]

You are God sent. Exactly what I have been looking for

[@UnfoldDataScience]

THanks alot

[@out_aloud]

This is what I had been looking for.... Great and simplified content ! Awesome !

[@UnfoldDataScience]

Thanks Nilut.

[@beyou7893]

thank you..........sirji..........what i saw from, video is recorded at 1:52am....shows the damn dedication...🙏

[@UnfoldDataScience]

Thanks for watching.

[@DimiqBaba]

My man recorded this at 2 AM to bring us content, much love

[@santosh44kumar]

Hi Aman, video time at: 5:12, the graph is +ve / right skewed, u stated its left skewed? any inputs on it appreciated.

[@UnfoldDataScience]

Thanks for inputs I will check, sometimes I get confused between left and right. In general also🤣

[@whatsuppk6466]

​@Unfold Data Science hahah its true.. i remember it by using position of outliers.. if outliers are on left, its left skewed.. if outliers on right, its right skewed

[@anuragchandnani8037]

Hi aman , I read this concept so many times but never understood it like I did today.. thanks

[@UnfoldDataScience]

Most Welcome Anurag. Your comments keep me motivated :)

[@dev5289]

Seeing this video before the interview.Thanks a lot

[@UnfoldDataScience]

Welcome Devansh.

[@maaleem90]

And brother, what was the result. I hope you got in

[@shubhamchoudhary5461]

straight forward explanation .. literally amazing.. i want to learn teaching skill from you sir.. thanks!

[@UnfoldDataScience]

Welcome Shubham.

[@APARNASHAHARE99]

hello sir, while performing shapiro test in output i am getting nan and 1 value for stats and p . can you please tell what could be the reason?

[@mukeshkumar-kh2fh]

far better explanation than any other youtube channel....awsm work Aman

[@UnfoldDataScience]

Thanks alot Mukesh.

[@mukeshrar265]

Thank you so much, Aman, for making hypothesis testing very simple

[@UnfoldDataScience]

My pleasure Mukesh :)

[@k10-w3s]

In chi Square test sub categories in features one should be equal to sub categories of feature two or can be any ? exp: feature 1 : pass ,fail , grace feature 2 : study , no study , mixed or exp: feature 1 : pass ,fail , grace feature 2 : study , no study , Which one is correct ?

[@anuradhabalasubramanian9845]

Thanks for a simplified explanation Sir

[@UnfoldDataScience]

Welcome.

[@randommagic3073]

This is what i was looking for and suddenly it appeared on my feed yesterday. I watched it and feeling blessed. Thanks for this content. Can i add this project on my resume anyhow if yes what would be title and description??

[@YashpalNSharma]

Very simple and effective summary of various hypothesis tests and statistical thinking. Thank you!

[@UnfoldDataScience]

Thanks Yashpal :)

[@YashpalNSharma]

Unfold Data Science I have always had very basic question. How do we get to decide the Null Hypothesis statement? It could also be the exact opposite of the current statement - and the results will be reversed as well. I think I’m missing something basic but unsure what that is. Please clarify. Thanks in advance!

[@UnfoldDataScience]

Very good question, this shows u r not satisfied until you understand the concept well. Coming to the answer, let's take "T-Test", when u go to the documentation of the package for example scipy. Ttest in python, u will clearly see it's written that " Null assumption is means are equal" Just an example, similarly for other packages and tools as well null hypothesis is mentioned.

[@YashpalNSharma]

Unfold Data Science Sure Aman. Will check it out. Thank you for clarifying.

[@jyotigoyal2295]

very nice explanation sir...

[@UnfoldDataScience]

Thanks Jyoti.

[@nayemhasan5015]

simple and useable. Great video.

[@bennurakash]

Thank you for the data science notes in python.

[@UnfoldDataScience]

Welcome :)

[@yogeshbharadwaj6200]

Very nice explanation with perfect examples, thanks a lot.....

[@UnfoldDataScience]

Most welcome Yogesh

[@santoshvaidya3752]

Aman - very good, keep it up

[@UnfoldDataScience]

Thanks Santosh.

[@ramakrishna1008]

Great explanation, simple and straight to the point. Can you please share the jupyter note book shown in this video.

[@letsplay0711]

9:32 , How you have deciced value above 0.85 and below -0.85 is correlated? Please respond...

[@dnyanjal5471]

You have not mentioned How to state which is null hypothesis and which is alternate hypothesis? Please explain this.

[@shivanshjayara6372]

sir by saying that 'observations are identical...can we also say that their dispersion is same from mean? or by that statement do we mean that it is already assume that data is normally distributed by default.....that what null hypothesis is?

[@kartikmeher8061]

Thank you🙏🙏🙏

[@UnfoldDataScience]

Welcome Kartik

[@shreyasb.s3819]

Very good explaination. Thanks

[@UnfoldDataScience]

Glad it was helpful Shreyas!

[@ArpitYadav-ws5xe]

Excellent Aman. You are excellent. Keep making more videos. Keep it up

[@UnfoldDataScience]

Thank you so much Arpit 😀

[@sudipnarayanchoudhury1112]

@Aman, What is that 'stat' output signifies? We are getting two results as the output of the tests one is p value another is stat(like stat=0.97 & p=0.0047). what is that stat value??

[@harshmohan8419]

I AM not able to understnd, whemn we have linar regression Ml models to calculate R square and other parameters.. why would we calculate the spearman cofficient... why would test, when ML model show lots paramters precis confusion matrix... Im bit confuse any guidance will helpful

[@ravneetkaur7278]

PLEASE reply to these queries- These misconceptions must get cleared as these foundational concepts of Data Science. 1. In Pearson correlation you are checking p value here but in last video u said we DO NOT CHECK P value instead we check closeness of 2 variables on the scale of -1 to 1. 2. Normality test is just to check the SHAPE -if the data is normally distributed or not ? What is the USE of P value here? How is checking P value in Shapiro beneficial? what is null hypothesis here ? Didn't understand. 3. Didn't understand DIFFERENCE in Shapiro test and k^2 normality test. What are VARIABLE TYPE here Categorical, Continuos etc what? 4. Pearson and Spearman are just 2 WAYS of checking correlation? Any of these can be chosen randomly to check correlation ? Or are there any SPECIFIC RULES to choose? 5. "value>0.85 on neg side , value below 0.85 s considered big " you spoke at 9:58th min FROM WHERE are we taking 0.85 ? what is its significance? Didn't understand. 6. Moreover, correlation is applied on CONTINUOUS vs CONTINUOUS data ONLY and measured from -1 to 1 whereas chi sq is applied on CATEGORICAL vs CATEGORICAL to check P value then WHY are we COMBINING chi sq and correlation when working on categorcal vs categorical.. It's confusing. Thanks in Advance!!

[@ravanshyam7653]

sir ,sharply at the time of 5:13 u telled that graph is left skewed actually it is right skewed am right???

[@UnfoldDataScience]

Yes, I sometimes confuse between left right may be you are right, thank you.

[@umashankarverma3179]

Very good explanation sir 👌

[@UnfoldDataScience]

Keep watching Uma.

[@techinfo89]

great explanation.. how to test null hypothsis

[@divyamsaxena3198]

can we have hypothesis testing practice session on different data please...........

[@datascienceworld7041]

Basically P value is used to check if the null hypothesis is true or not ??? Is that correct

[@UnfoldDataScience]

Yes Amrit

[@datascienceworld7041]

And its aim is to reject the null Hypothesis and Accept the Alternate Hypothesis??

[@varshakotwal3726]

how we can do not normal distribution to narmality on spss

[@souravbiswas6892]

Awesome explanation 👍👍

[@UnfoldDataScience]

Thanks again Sourav. Your comments are my power engine.

[@souravbiswas6892]

You are creating the video at 2am which clearly shows your dedication and love towards data science. Fully appreciated.

[@dr.vinodkamble4618]

For which kind of data we need to do normality test?

[@sagarmestry5514]

Aman I've one doubt like do we have to do hypothesis testing on every problem statement? Or have to use for just regression based to check the assumptions?

[@UnfoldDataScience]

Hi Sager, ideally yes to understand more about your data.

[@NinadNakhwa]

Hi Aman, Is it a Left skewed data or Right Skewed ?

[@UnfoldDataScience]

Hi Ninad, Ahh I think I might hv said other way, can you read previous comments please, I think I hv clarified this.

[@satyaki44]

Great playlist Aman Bhaiya ! Please keep up your good work..

[@UnfoldDataScience]

Thank you, I will

[@geekyprogrammer4831]

This is content is not widely available. The way you simiplied complicated concept is remarkable!

[@UnfoldDataScience]

Thanks a lot. Your comments keep me going.

[@quamar0313]

Thanks bro Please make video on what to do after these testing that we got some useful information about the dataset what next

[@UnfoldDataScience]

As soon as possible

[@pramod3469]

Hi Aman please explain where and why we use all these statistical test in machine learning model and its use

[@UnfoldDataScience]

Answered below.

[@rsinh3792]

Sir reviewer has asked me this question I don't know how to address it, can you please guide me "Use some statistical significant test such as T-test or ANOVA to prove you validate the proposed diagnostic model on patients and quality improvements of your method". I have two datasets. Dataset 1 was used to train the model and dataset 2 was used to validate the trained model. I have trained the ML model deployed it and Validated it on new data and presented the results. Actually, I have understood the question. Shall I apply the statistical test between the performance metrics of trained model results and validation results? Please help me, sir.

[@jayashreehv5222]

hi Aman, for correlation why not only heatmap or correlation table sufficient, why it should be tested again with P values or stats techniques

[@UnfoldDataScience]

heatmap becomes difficult to analyze when u have more features.

[@vishalrai2859]

sir re arrange the playlist part 2 is coming before part 11

[@shriyagoswami8452]

Can you explain what is stat value?

[@UnfoldDataScience]

Stat value, which part of video I spoke abt it? Just to correlate

[@shriyagoswami8452]

@@UnfoldDataScience stat,p=spearmanr(data1,data2) what is this stat ??

[@RohanB-xg6vg]

What is the stat variable returned by those fuctions. Stat, pvalue = spearmanr(firstsample,secondsample)

[@UnfoldDataScience]

Please refer here for detailed explanation: docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html

[@fahadnasir1605]

Thank You for sharing this amazing tutorial. Will you be able to share the CSV file of the data you have used?

[@UnfoldDataScience]

In the description

[@spicytuna08]

what does it mean when cat variables are dep or indep? i understand when 2 continuous variables are dep and indeo,

[@UnfoldDataScience]

cat variables will be dependent or target in case of classification scenarios

[@vishaldas6346]

Hi Aman, I owe you a big one, this is the best explanation ever for hypothesis testing. I have a doubt suppose I have 2 categorical variables of 2 features and they are dependent, so in case of feature selection can we drop 1 feature?

[@UnfoldDataScience]

Thanks Vishal. Yes for testing one variable at a time

[@ramshaazeemi8851]

How will we decide if p>0.05 then its an independent variable or dependent ?

[@UnfoldDataScience]

Depends on what hypothesis we have taken.

[@kalaisiva7437]

@@UnfoldDataScience Sir, please provide some examples for P>0.05. I am having same confusion.

[@parthbisht5597]

Sir what if we want to plot a correlation test on more than 2 independent categorical feature, please answer.

[@abhishekgaurav7786]

bro can we use correlation fun for categorical feature

[@UnfoldDataScience]

for anything more than 2d, a 3d chart is needed.

[@UnfoldDataScience]

Not pearson correlation.

[@aa4734]

Hi sir, please help me with deep learning and NLP notes as I'm not getting clear way...

[@UnfoldDataScience]

You can find lot of content on web

[@chetangondaliya]

Thank you so much for explanation. can u provide dataset which you used for this experiment?

[@UnfoldDataScience]

drive.google.com/drive/folders/1XdPbyAc9iWml0fPPNX91Yq3BRwkZAG2M

[@chetangondaliya]

​@@UnfoldDataScience This contain only Jupyter file. I want csv file which you used for date reading.

[@r21061991]

Great video.. Can you also make a video on the math behind all the tests..Would be great.. !!

[@dracula5505]

Sir what to do when one columns is categorically and another is numerical????

[@UnfoldDataScience]

Anova will work.

[@riteshtripathi8626]

hi, would appreciate if you could share the loan_status dataset that you have used in hypothesis testing (checked in your google drive as well), as i have searched for the exact in kaggle, but looks to me different version, thanks

[@UnfoldDataScience]

Hi Ritesh, hope u are doing good and staying safe with your near and dear ones. See if you find data here: drive.google.com/drive/folders/1XdPbyAc9iWml0fPPNX91Yq3BRwkZAG2M

[@lekshmia.k.8264]

Thanks a lot for the video Aman. I have a doubt what to do if I have less than 25 data for a categorical-categorical variables? If chi squared test didn't give the correct values, is there any other test which I can use in that case?

[@UnfoldDataScience]

Fisher's exact test

[@praneethaluru2601]

Sir, can you make a video on why these tests are used in machine learning and which algorithms get affected by this and how the affect would happen.

[@UnfoldDataScience]

Sure Praneeth, Thanks for suggesting. Noted.

[@bsarath438]

​@@UnfoldDataScience Sir,please provide a link for above comment if u done a video

[@dataartist5195]

hello sir your playlist is not arranged sequentially

[@UnfoldDataScience]

Let me check

[@sowmyatushar7487]

It is right skewed not left skewed. In the Video u mentioned it as left skewed.

[@UnfoldDataScience]

Yes sometimes I confuse just like left/right on road. Sorry If I said left skewed.

[@Amansingh-tr1cf]

u saved my ass..thanks Aman

[@UnfoldDataScience]

Cheers Sir Jee :)

[@tariqanwar8097]

Sir, do more on hypothesis test in practical

[@UnfoldDataScience]

Sure Tariq. More explanation in Part 2 is on the way.

[@2galacticos]

Aman, thank you for clear and precise explanation of Hypothesis testing. I wanted to ask how often do you use Hypothesis tests in your data to day EDA. I have seen many courses where in EDA we are only doing univariate, bi and multivariate analysis, but, no one explains the use of hypothesis testing in EDA.

[@UnfoldDataScience]

Hi Abhishek, thats a good question. Hypothesis testing is lot of hard work hence sometimes part of it are skipped however more you do, better you will know your data.

[@hoangha6680]

the code "from numpy.random import randn' doesn't always generate normally distributed data. I tried and the test result is nor normally distributed. But thanks for the video anw

[@UnfoldDataScience]

Thanks for the info!

[@akar_excel]

what is normal distribution?

[@UnfoldDataScience]

A normal distribution, also known as the Gaussian distribution, is a continuous probability distribution characterized by its bell-shaped curve. It is fundamental in statistics and is often used to represent real-valued random variables with a distribution that is symmetric around the mean.

[@ragsgags5901]

Please share loan_status.csv file, not available in drive

[@shilashm5691]

It is right skewed

[@UnfoldDataScience]

Ok

[@harshkashyap7822]

this video worth a value of a paid live class, because they charge 50K for the entire course

[@samratshimpi5093]

It is the best video I have seen so far for hypothesis testing. My name is Ameya Girhe. I have sent you request on LinkedIn. could you please accept?

[@UnfoldDataScience]

Thanks samrat. I will accept.

[@PraveenKumar-kc3ge]

sir provide notebooks of this video

[@sandipansarkar9211]

finisahe watching

[@UnfoldDataScience]

Thanks sadipan for following my videos in proper order. Sure you will get topics understood.

[@sandipansarkar9211]

@@UnfoldDataScience I talked with a friend Sourav Biswas and he said he knows you.You live in Bangalore (Varthur).I am also in Bangalore(Kadugodi).He is also in Bangalore

[@cocgamingstar6990]

it right skew data

[@UnfoldDataScience]

My mistake. I accepted it before in comments I believe

[@sagarlokare5269]

Confusing

[@UnfoldDataScience]

Thanks for feedback Sagar. will have alook in video