Classification Trees in Python from Start to Finish

[@statquest]

NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/tzxoh Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@ezzouaouia.r1127]

The site is offline. 11/07 12:00

[@statquest]

Thanks for the note. It's back up.

[@ezzouaouia.r1127]

@@statquest Thanks very much .

[@dfinance2260]

Still offline unfortunately. Would love to check the code.

[@statquest]

@@dfinance2260 It should be back up now.

[@funnyclipsutd]

BAM! My best decision this year was to follow your channel.

[@statquest]

BAM! :)

[@mike19558]

Mood, been so useful!

[@renekokoschka707]

I just started my bachelor thesis and i really wanted to thank you! Your videos are helping me so much. You are a LEGEND!!!!!

[@statquest]

Thank you and good luck! :)

[@user-lc8gc6vb3j]

Thank you, this video helped me a lot! For anyone else following along in 2023, the way the confusion matrix is drawn here didn't work for me anymore. I replaced it with the following code: cm = confusion_matrix(y_test, clf_dt_pruned.predict(x_test), labels = clf_dt_pruned.classes_) disp = ConfusionMatrixDisplay(confusion_matrix = cm, display_labels=['Does not have HD', "Has HD"]) disp.plot() plt.show()

[@statquest]

BAM! Thank you. Also, I updated the jupyter notebook.

[@jahanvi9429]

You are so so helpful!! I am a data science major and your videos saved my academics. Thank you!!

[@statquest]

Happy to help!

[@ravi_krishna_reddy]

I was searching for a tutorial related to statistics and landed here. At first, I thought this is just one among many low quality content tutorials out there, but I was wrong. This is one of the best statistics and data science related channels I have seen so far, wonderful explanation by Josh. Addicted to this channel and subscribed. Thank you Josh for sharing your knowledge and making us learn in a constructive way.

[@statquest]

Thank you very much! :)

[@joaomanoellins2219]

I loved your Brazil polo shirt! Triple bam!!! Thank you for your videos. Regards from Brazil!

[@statquest]

Muito obrigado!!!

[@cindinishimoto9528]

@@statquest paying homage to Brazil!!

[@statquest]

@@cindinishimoto9528 Eu amo do Brasil!

[@kaimueric9390]

I actually think it can be great if you created more videos for other ML algorithms. After teaching us almost every aspect of machine learning algorithms as far as the mechanics and the related fundamentals are concerned, I feel it is high time to see those in action, and Python is, of course, the best way to go.

[@statquest]

I'm working on them!!! :)

[@juniotomas8563]

Come on, Buddy! I've just saw a recommendation to your channel and on the first video I see you with a Brazilian t-shirt. Nice surprise!

[@statquest]

Muito obrigado! :)

[@pratyushmisra2516]

My intro song for this channel: " It's like Josh has got his hands on python right, He teaches Ml and AI really Well and tight ---- STAT QUEST" btw thanks Brother for so much wonderful content for free.....

[@statquest]

Thank you! :)

[@Mohamm-ed]

This voice remembering me when I listening to radio in UK. Love that. I want to go again

[@statquest]

:)

[@ozzyfromspace]

I dunno how I stumbled on your channel a few videos ago, but you've really got me interested in statistics. Nice Work sir 😃

[@statquest]

Hooray!

[@liranzaidman1610]

Josh, this is really great. Can you upload videos with some insights on your personal research and which methods did you use? And some examples of why you prefer to use one method instead of the other? I mean, not only because you get a better result in RUC/AUC but is there a "biological" reasoning for using a specific method?

[@statquest]

Great suggestion!

[@creativeo91]

This video helped me a lot for my Data Mining assignment.. Thank you..

[@statquest]

Glad it helped!

[@ccuny1]

I have already commented but I watched the video again and I have to say I am even more impressed than before. truly fantastic tutorial, not too verbose but with every action clarified and commented in the code, beautifully presented (I have to work on my markdown; there are quite a few markdown formats you use that I cannot replicate...to study when I get the notebook). So all in all, one of the very top ML tuts I have ever watched (including paid for training courses). Can't wait for today's or tomorrows webinars. Can't join in real time as based in Europe, but will definitely pick it up here and get the accompanying study guides/code.

[@statquest]

Hooray!!! Thank you very much!!!

[@dhruvishah9077]

I'm absolute beginner and this is what i was looking. Thank you so much for this. Much appreciated sir!!

[@statquest]

Glad it was helpful! :)

[@montserratramirez4824]

I love your content! Definitely my favorite channel this year Regards from Mexico!

[@statquest]

Wow, thanks! Muchas gracias! :)

[@1988soumya]

Hey Josh, it’s so good to see you are doing this, I am preparing for some interviews, it will help a lot

[@statquest]

Good luck! :)

[@ccuny1]

Another hit for me. I will be getting the Jupyter notebook and some if not all of you study guides (I only just realised they existed).

[@statquest]

BAM! :) Thank you very much! :)

[@korcankomili7398]

I wish you were my uncle Josh or something. I could imagine how hard I would have had discussions with my parents to spend time with my TRIPLE cool uncle.

[@statquest]

bam! :)

[@bressanini]

Hey Josh, follow this equation: You + Brazilian Flag Polo Shirt + Awesome Content = TRIPPLE BAM!!!

[@statquest]

Muito bem! :)

[@anishchhabra5313]

This is legen..... wait for it ....dary!! 😎 This detailed coding explanation of Decision Tree is hard to find but Josh you are brilliant. Thank you for such a great video.

[@statquest]

Glad you liked it!

[@rajatjain7465]

wowowowwo the best course ever, even better than all those paid quests thank you @josh stramer for these materials

[@statquest]

Thank you! :)

[@jefferyg3504]

You explain things in a way that is easy to understand. Bravo!

[@statquest]

Thank you! :)

[@sameepshah3835]

I love you so much Josh. Thank you so much for everything.

[@statquest]

Thanks!

[@Moiez101]

1 hour statquest? in the words of Barney Rubble's son: "BAM BAM!"

[@statquest]

double bam! :)

[@willw4096]

1:00:20 Use color to visualize the category and the Gini impurity

[@breopardo6691]

As Tina Turner would say: "You are simply the best!" 🎵🎵🎵

[@statquest]

BAM! :)

[@aleksandartta]

How to implement pipeline with cost complexity? Consider the marking part which start before 49:00... Thank in advance! You are the best teacher...

[@statquest]

I'll work on that

[@DANstudiosable]

OMG... I thought you'd ignore when i asked you to post this webinar on youtube. Am glad you posted it. Thank you!

[@statquest]

BAM!!!!

[@shubhankarde4732]

Double BAM

[@bessa0]

Kind Regards from Brazil. Loved your book!

[@statquest]

Thank you!

[@gbchrs]

your channel is the best at explaining complex machine learning algorithm step by step. please make more videos

[@statquest]

Thank you very much!!! Hooray! :)

[@bayesian7404]

You are fantastic! I'm hooked on your videos. Thank you for all your work.

[@statquest]

Glad you like them!

[@beebee_0136]

I'd like to thank you so much for making this stream cast available!

[@statquest]

:)

[@aryamohan7533]

This entire video is a triple bam! Thank you for all your content, I would be lost without it :)

[@statquest]

Glad you enjoyed it!

[@lawrencegayundato8398]

@@statquest This is Quadruple BAM!!!! Thank you Mr. Josh :)

[@sharmakartikeya]

Hurray! I saw your face for the first time! Nice to see one of those whom I have subscribed

[@statquest]

bam!

[@filosofiadetalhista]

Loved it. I am working on Decision Trees on my job this week.

[@statquest]

bam!

[@randyluong6275]

We have data scientist out there. We have "data artist" right in this video.

[@statquest]

Wow! Thank you!

[@JoRoCaRa]

brooo... this is insane!! thanks so much! this is amazing saving me so many headaches

[@statquest]

Glad it helped!

[@3ombieautopilot]

Thank you very much for this one! You're channel is incredible! Hats off to you

[@statquest]

Bam! :)

[@fuckooo]

Love your videos Josh, the notebook missing values sounds like a great one to do!

[@statquest]

Awesome!

[@magtazeum4071]

BAM...!!! I'm getting notifications from your channel again

[@statquest]

BAM! :)

[@amc9520]

Thanks for making my life easy.

[@statquest]

Any time!

[@josephgan1262]

Hi Josh, Thanks for the video again!!. I have some questions hope you don't mind to clarify in regards to pruning in general hyperparameter tuning. I see that in general the video has done the following to find the best alpha. 1) After train test split, find the best alpha after comparison between test and training (single split). @50:32 2) Rechecking the best alpha by doing CV @52:33. It is checked that that is huge variation in the accuracy, and this implies that alpha is sensitive to different training set. 3) Redo the CV for to find the best alpha by taking the mean of accuracy for each alpha. a) At step two, do we still need to plot the training set accuracy to check for overfitting? (it is always mention that we should compare training & testing set accuracy to check for overfitting) but there is an debate on this as well. ( Where other party mentioned that for a model-A of training/test accuracy of 99/90% vs another model-B : 85/85%. We should pick model-A with 99/90% accuracy because 90% testing accuracy is higher than 85% even though the model-B has no gap (overfitting) between train & test. What's your thought on this? b) What if I don't do step 1) and 2) and straight to step 3) is this a bad practice? do i still need to plot the training accuracy to compare with test accuracy if I skip step 1 and step 2? Thanks. c) I always see that the final hyper parameter is decided on highest mean of accuracy of all K-folds. Do we need to consider the impact of variance in K-fold? surely we don't want our accuracy to jump all over the place if taken into production. if yes, what is general rule of thumb if the variance in accuracy is consider bad. Sorry for the long posting. Thanks!

[@statquest]

a) Ultimately the optimal model depends on a lot of things - and often domain knowledge is one of those things - so there are no hard rules and you have to be flexible about the model you pick. b) You can skip the first two steps - those were just there to illustrate the need for using cross validation. c) It's probably a good idea to also look at the variation.

[@utkarshsingh2675]

this is what I have been looking for on youtube...thanks alot sir!!

[@statquest]

Thanks!

[@nataliatenoriomaia1635]

Great video, Josh! Thanks for sharing it with us. And I have to say: the Brazilian shirt looks great on you! ;-)

[@statquest]

BAM! :)

[@rhn122]

Great tutorial! One question, by looking at the features included in the final tree, does it mean that only those 4 features are considered for prediction, i.e., we don't need the rest so we could drop those columns for further usage?

[@statquest]

That is correct.

[@xiolee7597]

Really enjoy all the videos! Can you do a series about mixed models as well, random effects, choosing models, interpretation etc. ?

[@statquest]

It's on the to-do list.

[@junaidmalik9593]

Hi Josh, one amazing thing about the playlist is the song u sing before starting the video, that refreshes me. u know how to keep the listener awake for the next video. hehe. and really thanks for the amazing explanation.

[@statquest]

Awesome thank you!

[@floral7448]

Finally have the honor to see Josh :)

[@statquest]

:)

[@rahulthaker694]

You look exactly how I thought you'd look like 😂

[@statquest]

BAM! :)

[@rahulthaker694]

@@statquest yesss legend 😂🙏

[@statquest]

@@rahulthaker694 :)

[@chrissmith1152]

he's done a lot of Live stream previously

[@BeSharpInCSharp]

I think he looks like joshua from Friends ( friend of rachel)

[@robertmitru7234]

Awesome StatQuest! Great channel! Make more videos like this one for the other topics. Thank you for your time!

[@statquest]

Thanks! Will do!

[@jonastrex05]

Amazing video! One of the best out there for this Education! Thank you Josh

[@statquest]

Thank you!

[@mahdimj6594]

Neural Network Pleaseee, Bayesian and LARS as well. And Thank you. You actually make things much easier to understand.

[@statquest]

Thanks! :)

[@simaykazc1508]

Josh is the best. I learned a lot from him!

[@statquest]

Wow! Thank you!

[@naveenagrawal_nice]

Love this channel, Thank you Josh

[@statquest]

Glad you enjoy it!

[@douglasaraujo9763]

Your videos are always very good. But today I’ll have to commend you on your fashion choice as well. Great-looking shirt! I hope you have had the opportunity to visit Brazil.

[@statquest]

Muito obrigado! Eu amo do Brasil! :)

[@SamirMishra6174]

wow is that a tabla in the background ?

[@statquest]

Yes! I used to play tabla a long time ago.

[@SamirMishra6174]

@@statquest Amazing you are multi talented.

[@fernandosicos]

greatings from Brazil!

[@statquest]

Muito obrigado! :)

[@ramendrachaudhary9784]

We need to see you play some tabla to one of your songs. Double BAM!! Great content btw :)

[@statquest]

Maybe one day!

[@amalsakr1381]

Thank you for your powerful tutrial

[@statquest]

Glad it was helpful!

[@_ahahahahaha9326]

Really learn a lot from you

[@statquest]

Thanks!

[@umairkazi5537]

Thank you very much . This video is very helpful and clears a lot of concepts for me

[@statquest]

Bam! :)

[@ericwr4965]

I absolutely love your videos and I love your channel. Thanks for this.

[@statquest]

Thanks! :)

[@paulovinicius5833]

I know I'll love all the content, but I start liking the video immediatly bc of the music! haha

[@statquest]

Thank you! :)

[@jihowoo9667]

I really love your video, it helps me a lot!! Regards from China.

[@statquest]

Awesome! Thank you!

[@estebannantes8567]

Hi Josh. Loved this video. I have two questions: 1- Is there any way to save our final decision tree model to use it later in unseen data without having to train it all again? 2- Once you have decided on your final alpha: why not training your tree on a full-unsplit dataset. I know you will not be able to generate a confusion matrix, but wouldn't your final tree be better if it is trained with all the examples?

[@statquest]

Yes and yes. You can write the decision tree to a file if you don't want to keep it in memory (or want to back it up). See: scikit-learn.org/stable/modules/model_persistence.html

[@lucillewiid5476]

Hi, Josh, recommend your videos to all my students and love watching and learning from them 👍. Can we still download this notebook?? Or do we need to buy it?? Regards from South Africa!

[@statquest]

This notebook has always been for sale and is still for sale if you would like it.

[@liranzaidman1610]

Fantastic, this is exactly what I needed

[@statquest]

Hooray! :)

[@chaitanyasharma6270]

i loved your video support vector machines in python from start to finish and this one too!!! can you make more on different algorithms?

[@statquest]

I will try!

[@avramisthename]

great insight and refresher, thank you for documenting

[@statquest]

Glad you enjoyed it!

[@Kenwei02]

Thank you so much for this tutorial! This has helped me out a lot!

[@statquest]

Glad it helped!

[@abdelrhmansayed5436]

thank you for your great effort and simple explanation, i have only one question that is why did you split the data into X_train and y_trrain and then give it to cross_val_score , shouldn't coss validtion works on all X ?

[@statquest]

In theory we are trying to save some data for a final validation of the model.

[@teetanrobotics5363]

Amazing man. I love your channel. Could you please reorder this video , SVMs and Xgboost in the correct order in the playlist ?

[@statquest]

Yes!

[@vipanpatial2243]

BAM!! You are best.

[@statquest]

Double bam!

[@mcmiloy3322]

Really nice video. I thought you were actually going to implement the tree classifier itself, which would have been a real bonus but I guess that would have taken a lot longer.

[@statquest]

Noted

[@saiakhil4751]

Wow!! Josh on live? made my day...

[@statquest]

:)

[@ayatkhrisat5964]

kindly add this video to the machine learning list

[@statquest]

Will do!

[@srmsagargupta]

Thank you Sir for this wonderful webinar

[@statquest]

Thank you!

[@michelchaghoury870]

MANNNN so usefull please keep going

[@statquest]

Thanks!

[@amitsaxena6530]

Hi Josh, Request you to make more such ML videos in python which covers all ML concepts holistically. I am sure this course will then become more popular then any of the available ML courses. Pls pls pls....

[@statquest]

I'll consider it.

[@TalesLimaFonseca]

Man, you are awesome! Vai BRASIL!!!

[@statquest]

Muito obrigado!

[@prnv5]

Hi Josh! I'm a HS student trying to learn ML algorithms and your videos are genuinely my saving grace. They're so concise, information heavy and educational. I understand concepts perfectly through your statquests, and I'm really grateful for that. One quick question: The algorithm used in this case to build a decision tree: is it the CART algorithm? I'm writing a paper on the CART algorithm and would hence like to confirm the same. Thanks again!

[@statquest]

Yes, this is the "classification tree" in CART.

[@prnv5]

@@statquest Thank you so much 🥰

[@kaimueric9390]

I liked before watching

[@statquest]

Hooray!!!

[@krishanudebnath1959]

love the tabla and ur content

[@statquest]

Thanks! My father used to teach at IIT-Madras so I spent a lot of time there when I was young.

[@alpatul]

This is great, do you have any more python webinars related to machine learning? I would love to go through them.

[@statquest]

I'm working on more right now.

[@oliveryoule11]

@@statquest At 19 minutes you say you have plans for a whole webinar on missing data! This is what I need. Where can I find it or is it still in production? :D

[@statquest]

@@oliveryoule11 Dang! I'd forgotten about that. I guess you could say it's still in production. :)

[@oliveryoule11]

​@@statquest Thanks for replying! I can see how easy it is to forget! You have so much content its unreal! V impressive! I just purchased your Notebook through the link - but it doesn't appear to arrived in my inbox. Can you advise? I am also strongly considering paying for your Patreon account. I currently pay for Datacamp - but your material is so much better!

[@statquest]

@@oliveryoule11 Wow! Thanks for supporting me and I'm sorry you had trouble purchasing the notebook. If you contact me through my website, I can send it tor you directly: statquest.org/contact/

[@DanteNoguez]

Double BAM! Haha, I love this guy

[@statquest]

Thank you!

[@PinkFloydTheDarkSide]

Somehow your room and furniture remind me of my grad building room at the Univ. of Chicago.

[@statquest]

Cool!

[@shindepratibha31]

I have almost completed the Machine learning playlist and it was really helpful. One request, can you please make a short video on 'handling the imbalanced dataset'?

[@statquest]

I've got a rough draft on that topic here: kzbin.info/www/bejne/n4Xbq4WMgdSHh5I

[@Denise_lili]

Thank you Josh for the nice videos! Questions: 1) What is accuracy? Is there a relationship between Gini impurity/Sum of squared residuals and accuracy (i.e. Lower Gini impurity means higher accuracy)? 2) Once we create a tree classifier with a certain alpha, will different training data sets give different fitted trees? And how are they different?

[@statquest]

Accuracy is the percentage of the data that are correctly classified. The lower the gini index, the higher accuracy. Different training datasets will probably give a different value for alpha, so it's good to use cross validation to find the best value.

[@Denise_lili]

@@statquest Thank you Josh for the quick reply. This is very helpful! But AFTER the optimal alpha value is identified via the cross validation, will different training datasets give different final pruned trees in the below query? clf_dt_pruned = clf_dt_pruned.fit (X_train, y_train)

[@statquest]

@@Denise_lili Yes

[@rogertea1857]

Pruning is better than setting max_depth or min_samples beforehand overall I guess. Thanks for another great tutorial : ）

[@statquest]

Thanks!

[@bitseatomos]

Congratulations ! Ten times triple bam !!

[@statquest]

Hooray! :)

[@Mustistics]

Hey Josh. One thing that bugs me about this tutorial: when you do binary classification, you need to take into account class imbalance. Accuracy is the worst metric for this. Was that neglected for a reason?

[@statquest]

No, ideally we would take class imbalance into account.

[@Mustistics]

@@statquest Thanks for the response. So theoretically, I can follow your tutorial exactly, but use recall wherever you use accuracy?

[@statquest]

@@Mustistics Sure!

[@Mustistics]

@@statquest You're the best (and I'm not just saying that)!

[@GokulSKumar-uz9dy]

Great video sir.:) I just have a doubt in one part. At 52:14 instead of using X_train and y_train, arent we supposed to use the entire dataset(i.e. X_encoded and y) while implementing cross-validation? Also later in the video at 52:54, the value for alpha was found by using only X_train and y_train data in the cross-validation.

[@statquest]

There are different schools of thought about what datasets you should use for cross validation. The most common one, however, is to do it as presented in this video.

[@GokulSKumar-uz9dy]

@@statquest Thanks a lot! Just in case if it might be useful, I tried using the entire dataset in cross-validation before splitting it into train and test. I could see from the corresponding confusion matrix that the model predicted correctly 90% of people not having a heart disease whereas, there was no increase in the percentage of people having heart disease. Again loved the video a lot. Waiting for the next webinar.:)

[@KeigoEdits]

@@statquest Hey Josh sir, actually after reading this comment I really went for cross-validation with the whole dataset as in above comment I also read that you mentioned that we should take whole dataset in case of small datasets and what I personally think is 297 datapoints dataset can be called small. This gave me better results at alpha=0.021097 and it was varying from 0.74 and 0.88. What are your views on it?

[@statquest]

@@GokulSKumar-uz9dy It really depends on how noisy your data is and what you hope to do with it.

[@RAMAYATRI]

Can see Tabla in the background.... Planning to use it in any future video ?

[@statquest]

One day!

[@BeSharpInCSharp]

i wanted to learn DT from scratch but it seems here we should already know things like confusion matrix. I better study that first and come back to this video

[@statquest]

Yep.

[@HB-ys9rt]

You're a great professor. One point that I'd like to ask for your opinion is that once you created dummy variables for a categorical variable with several levels, you did not remove one of the dimensions from X. Do you think that multicollinearity would be causing a bias in the model?

[@statquest]

No. Classification trees are not effected by multicollinearity.

[@HB-ys9rt]

@@statquest thank you. You’re just incredible!

[@InternatoMiguel]

Hello Josh, thank you so much for another great video! Did you end up doing a webinar on inputting values? If so, where can I find it? :)

[@statquest]

Maybe. What time point in the video, minutes and seconds, are you asking about?

[@InternatoMiguel]

@@statquest 18:29

[@statquest]

@@InternatoMiguel Unfortunately, except for imputing data for Random Forests, I haven't covered that topic very much. However, if you are interested in how Random Forests do it... kzbin.info/www/bejne/qYKbaGOXibCkn68

[@raindrop0405070]

First, Thank you. You explain complicated things in very easiest way with visulization. But,You should have a better microphone with it. I think I am going to keep wathcing your videos.

[@statquest]

Thanks for the tips!

[@beibeima524]

Hi Josh, Thanks so much for the video! My Question is should we do one hot encoding before or after splitting the data into training and testing set? Thanks!

[@statquest]

As long as all categories are in both sets, it doesn't matter.