Selecting the best model in scikit-learn using cross-validation

In this video, we'll learn about K-fold cross-validation and how it can be used for selecting optimal tuning parameters, choosing between models, and selecting features. We'll compare cross-validation with the train/test split procedure, and we'll also discuss some variations of cross-validation that can result in more accurate estimates of model performance.
Download the notebook: github.com/justmarkham/scikit...
Documentation on cross-validation: scikit-learn.org/stable/module...
Documentation on model evaluation: scikit-learn.org/stable/module...
GitHub issue on negative mean squared error: github.com/scikit-learn/sciki...
An Introduction to Statistical Learning: www-bcf.usc.edu/~gareth/ISL/
K-fold and leave-one-out cross-validation: • Video
Cross-validation the right and wrong ways: • Video
Accurately Measuring Model Prediction Error: scott.fortmann-roe.com/docs/Me...
An Introduction to Feature Selection: machinelearningmastery.com/an-...
Harvard CS109: github.com/cs109/content/blob...
Cross-validation pitfalls: www.jcheminf.com/content/pdf/1...
WANT TO GET BETTER AT MACHINE LEARNING? HERE ARE YOUR NEXT STEPS:
1) WATCH my scikit-learn video series:
• Machine learning in Py...
2) SUBSCRIBE for more videos:
kzbin.info?su...
3) JOIN "Data School Insiders" to access bonus content:
/ dataschool
4) ENROLL in my Machine Learning course:
www.dataschool.io/learn/
5) LET'S CONNECT!
- Newsletter: www.dataschool.io/subscribe/
- Twitter: / justmarkham
- Facebook: / datascienceschool
- LinkedIn: / justmarkham

[@dataschool]

Having problems with the code? I just finished updating the notebooks to use *scikit-learn 0.23* and *Python 3.9* 🎉! You can download the updated notebooks here: github.com/justmarkham/scikit-learn-videos

[@giovannibruner8455]

This videos are so well done, so clear and easy to follow that it makes appear ML a trick for kids. Congratulations, great teaching.

[@dataschool]

Thanks for your kind words!

[@deneb6139]

can't agree more! best video resource on cross validation on the internet.

[@thevivekmathema]

i almost gave up python,untill i met your channel. you are my savior

[@dataschool]

Wow, thank you! Good luck with your Python education! :)

[@ngochua6679]

Kevin, I appreciate the slow but thorough walk through, you and StatQuests are awesome people. Thank you.

[@dataschool]

Thank you so much!

[@apachaves]

Amazing video! Very instructive. And the presenter has a very clear voice and pace.

[@dataschool]

Thank you so much! I'm glad you liked it!

[@datascienceds7965]

Whenever I needed a references, I always end up with your videos after a long search. That prove you are THE best teacher.

[@dataschool]

What a nice thing to say! Thank you! :)

[@atwinemugume]

I love the simplicity in the videos. Thank you. I have learned some things that were confusing before. Especially cross validation

[@dataschool]

Awesome! That's great to hear.

[@BrothersFreedive]

Excellent series! This is the first time I've studied machine learning. You are doing an outstanding job of transforming it from a science fiction term into a tangible subject. I really appreciate these videos!

[@dataschool]

BrothersFreedive You're very welcome! I greatly appreciate your kind comments!

[@cozylifemodular1863]

Just chiming in to thank you for the series, really helps demistify and fill in the gaps. Looking forward to working through

[@dataschool]

You're very welcome!

[@manasa41087]

I am so glad I found you. I am aspiring an data scientist and I find all your videos extremely useful and better than any documentation since you explain the intricate details very well. Thanks!!

[@dataschool]

Wow, thanks so much for your kind comments! Good luck in your journey to become a data scientist!

[@danishbhatia5004]

Sir, I really appreciate your work. I strongly think that these video series is probably the best I have ever come so far. I truly praise the way you teach, this becomes utmost clear. Thank you very much for these videos and hope to see more of them.

[@dataschool]

I'm glad to hear my videos have been helpful to you!

[@mueez.mp4]

HOW DOES THIS VIDEO NOT HAVE LIKE A MILLION VIEWS?!? So good. Thank you, man!

[@dataschool]

HA! Thank you :)

[@jamesdalley2394]

I watched a dozen videos on this topic. I was pretty certain I understood it, but I still had a few questions. You're video cleared those questions up amazingly! Thank you.

[@dataschool]

That's awesome to hear! 🙌

[@wlancer8826]

You're soooooo good at explaining confusing concepts!!! I'm always wondering about the negative sign in loss function until now!!! Thank you!!!

[@dataschool]

You're very welcome! You might also want to read this post for an update: www.dataschool.io/how-to-update-your-scikit-learn-code-for-2018/

[@jiwachhetri4165]

This is the best sklearn tutorial I have come across.

[@dataschool]

Thank you!

[@colmorourke4657]

Outstanding work once again Kevin. A treasure to newcomers in the area.

[@dataschool]

Thank you!

[@lucamarcello9696]

The best explanation of cross-validation on the internet. Thank you!

[@dataschool]

Thank you!

[@anilreddyk5]

Thanks for the Video. This is the best Video on KNN Cross validation that I have watched. Appreciate your effort...

[@dataschool]

Thanks! :)

[@hamkam33521]

I found a lot and lot of videos about ML and cross validation, I watched them all, I tried to follow but it was very hard understand. But you, you make it easier, I was very confused with this cross validation and now it's more than clear. Thank you very much for this video and for your channel

[@dataschool]

You're very welcome! Glad it was helpful to you!

[@djs749]

Domain of liking something always was dominated on BY spontaneity BUT never it was without reasons. I liked all your videos with much enthusiasm and the simple reason is they are just BRILLIANT!

[@dataschool]

Thank you so much!

[@davidtemael1307]

I took coursera lesson twice but never got what was going on and you bro walked me through like I've never expected! thank you

[@dataschool]

Awesome! You are very welcome!

[@edbull4891]

Always eager to learn. You demystified the subject and you even made it easy for a 75 year old brain to comprehend ML methods. :) :) :)

[@dataschool]

Great to hear!

[@johnathangonzalez5286]

REALLY well done!! I found this video extremely helpful. Keep up the good work

[@dataschool]

Thanks for your kind comment! :)

[@dataschool]

*Note:* This video was recorded using Python 2.7 and scikit-learn 0.16. Recently, I updated the code to use Python 3.6 and scikit-learn 0.19.1. You can download the updated code here: github.com/justmarkham/scikit-learn-videos

[@mubeenkhan8210]

Updated link shows me this message : Sorry, something went wrong. Reload?

[@MasterofPlay7]

is this still relevant 2020?

[@tush16ar]

being a beginner to machine learning , this video lecture series are of great help ,providing crystal clear understanding of the concepts presented along the course . Dear sir please keep up with the good work

[@dataschool]

That's great to hear! Good luck with your education!

[@miguelamaro4900]

i have watched several videos on this subject. this was the only one that has met my expectations

[@dataschool]

Thanks for your kind words!

[@XRobotexEditz]

The Best explanation I have seen ever.

[@dataschool]

Wow, thank you so much!

[@richardpacholski2715]

Thank you Kevin, This is fantastic material. You made it easy for 60 year old brain to comprehend ML methods. Cross-validation material was excellent. I will try to run it on my own data sets. Looking forward to the next one. Regards Richard

[@dataschool]

Richard Pacholski Awesome! Very glad to hear :) I'm looking forward to making the next one!

[@marvinjosephagor9493]

These are all great material. Thank you very much for uploading these videos. Keep up the great work and know that they are all appreciated! :)

[@dataschool]

+MJoseph A Excellent! You are very welcome.

[@akshaysingh1914]

Sir firstly I would like to you thanks a lot, because you spent so much time to make this video ....this is really helpful to initial phase learner's for ML ; keep doing sir , I stopped this video in mid to say thanks , you saved my lots of hour to understand cross validation.....

[@dataschool]

That's awesome to hear! Thanks so much for letting me know! 🙌

[@saisreenivas8875]

You are awesome...you teach everything in a simple way....ask for the feedback....and make them much better.....And the best thing is you make everything (I REPEAT EVERYTHING) easy for us....So sweet of you :)

[@dataschool]

That is so kind of you to say! Thank you so much 😄

[@prathameshmahankal4180]

I really love your videos. They are so simple and to the point! Thanks for making such videos. :)

[@dataschool]

Thanks very much for your kind words!

[@andretenreiro]

Great videos to learn about machine learning! Thanks Kevin for making this avaiable.

[@dataschool]

You're welcome!

[@soumyareddy3695]

Hi Kevin, You are such a great teacher. Love your videos. Cant thank you enough!!

[@dataschool]

Thanks very much for your kind words! You are very welcome :)

[@Bena_Gold]

This is the best explanation so far ... far better than my professor ... thumbs up ...

[@dataschool]

Great to hear! :)

[@juancarlosesquivel7855]

Very clear and detailed explanations. Also the links after the videos are very helpful. Thanks.

[@dataschool]

You're welcome!

[@marwanalbadawii]

Your explanations are very straightforward. Thanks a lot.

[@dataschool]

Thanks!

[@aditidubey2826]

amazingly explained. Sufficiently Slow for a fresher in Machine learning.. Easily understandable. Keep it up.

[@dataschool]

Awesome, thank you! :)

[@KS-ko2zl]

Thank you so much. Your tutorial and style of explaining is exceptional.

[@dataschool]

Thank you!

[@flamboyantperson5936]

Great videos. Please keep the good work doing. We really need your lectures. Thank you so much.

[@dataschool]

Thanks for your kind words! I will definitely release more videos! :)

[@flamboyantperson5936]

Waiting eagerly for new series because I have completed watching all your videos. Thank you so much for teaching me Python. You have made me educated you are a teacher for me and I respect you. Thank you so much.

[@dataschool]

Awesome! Thank you for watching and learning! :)

[@dawittekie3796]

I am so happy to follow such kind of lecture because your teaching way is attractive and your language clarity is very excellent so I get knowledge an input for my ML thesis because am doing on classification (prediction) problem

[@dataschool]

Glad to hear that my videos are helpful to you! Good luck with your thesis!

[@pierrelaurent8284]

Can't wait to see next lesson ! Bravo

[@dataschool]

Thanks!

[@luismiguelcrespo9499]

Had to listened to it in 1.5 speed but it is very clear and concise. Thanks.

[@dataschool]

Thanks!

[@stjepan_8902]

thank you for introducing me to ML, and also for helping me understand Python through your great pandas videos!

[@dataschool]

You are very welcome!

[@mightyhlungwane2639]

Kevin, you are very good in explaining. I wish I found you earlier. I just subscribed to receive all future videos and thank you for all the explanations.

[@dataschool]

Thanks for your kind words! 🙏

[@aawinecoff]

I really appreciate how easy this video series is to follow and that the notebooks are available so you can follow along. It would be excellent if the notebooks were updated to reflect that the cross_validation is now deprecated.

[@dataschool]

Thanks for the suggestion! Right now I'm on paternity leave from Data School, but it's on my to-do list :)

[@johnf9231]

Congratulations!

[@jsx0328]

It's deprecated, but it still works for me in Jupyter Notebook... I literally did the exact same cross validation

[@dataschool]

I recently updated the code to use Python 3.6 and scikit-learn 0.19.1. The updated code can be found here: github.com/justmarkham/scikit-learn-videos

[@michaelbmotlhabi]

All your videos W.I.N - you’re the best

[@rakeshkumarkuwar6053]

Thank you sir for such a detailed explanation. I was struggling with the topic. Then luckily found your video and very bit of it is full of knowledge. Thanks again for making such informative videos.

[@dataschool]

Great to hear!

[@raidtape123]

I really really appreciate your efforts..this series is so helpful in learning.

[@dataschool]

I'm glad to hear the series is helpful to you! :)

[@mdinesk]

These videos have been very useful and excellent! Thanks!

[@dataschool]

+Dinesh Kumar Murali Great, thanks for your kind words!

[@unsharma9229]

i usually don't subscribe any channel but you earned this subs from me...keep going lots of love

[@dataschool]

Thank you! 🙌

[@nikhilpandey2364]

Thanks. The notebook helped me a lot. I hope more topics get coverage like this one had. Please do a video on PCA.

[@dataschool]

Thanks for your suggestion! I'll consider it for the future.

[@jsbros.]

It was confusing at first but you made it so clear I wish you will upload many other videos on ML. If you will upload a application on ML with python will be great for every one. Thank you. Love your way of teaching.

[@dataschool]

Thanks for your kind words! Here is my series on machine learning with Python: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@ianmelanson9520]

Great videos. Good approach combining thought process and tools.

[@dataschool]

+Ian Melanson Thanks!

[@tabnaka]

Looking forward to the next video on this!

[@dataschool]

tabnaka Great! It will come out in about two weeks.

[@johnnovotny4286]

Excellent. Thanks for sharing your expertise.

[@dataschool]

Thank you!

[@nonyabeeswax]

Natural born teacher. Bravo! Thank you!

[@dataschool]

Thank you! :)

[@jnandikonda]

Best Explanation ever done by anyone in Machine Learning Community. Hats Off to your Great work and effort in teaching us. May god bless you. Yes we need more concepts on Scikit learn than pandas. but use pandas functionality when needed

[@dataschool]

Thanks very much for your kind words!

[@Renan-st1zb]

Great videos! It is well explained, once you understand why (and this is so important) you are using some function or model. Besides, you also have a great resource material (and it shows you have done it with excelence). You are an awesome professor! Congrats, from Brazil :)

[@dataschool]

Thanks so much for your kind words! I'm glad the videos have been helpful to you. Good luck with your machine learning education!

[@RHONSON100]

Just like Andrew Ng you are a genius....he gave a clear explanation in theory and you produced a mesmerising implementation techniques in such a simple way that is inexplicable..............Awesome machine learning video i have watched so far...you helped me a lot you could never imagine..thank you sir

[@dataschool]

Thanks very much for your kind words!

[@RHONSON100]

you are most welcome sir

[@salmanpatel5666]

Thanks a ton, perfectly explained the concept and the code

[@dataschool]

Great to hear!

[@pranayrungta]

Very well explained!!!! Explanation is very impressive...

[@dataschool]

Thanks!

[@eyalcarmi3984]

Your video tutorial is very good. Thank you for helping understand these topics

[@dataschool]

You're welcome!

[@Lala-qh3wl]

Great ! Thank you very much for sharing such a clear explanation 🙌

[@dataschool]

You're very welcome!

[@bevansmith3210]

Thanks Kevin, very very helpful, as always.

[@dataschool]

You're very welcome! Glad it was helpful to you :)

[@manishthapliyal6372]

Beautifully explained

[@dataschool]

Thanks!

[@mikezhu7852]

THE best tutorial ever! Thank u

[@abusaleham]

Awesome explanation....!

[@dataschool]

Thanks!

[@ovoalways]

Great job Kevin. You're videos are really helpful

[@dataschool]

Ogheneovo Dibie Glad you're enjoying them!

[@ovoalways]

Thanks Kevin. Do you know of any resources that make it easy to import and transform my own datasets before using sci kit learn? I have data samples contain both numerical, categorical and boolean features. Thanks Kevin

[@dataschool]

Ogheneovo Dibie I mostly use Pandas for data reading and transformation. I demonstrate Pandas in this video: kzbin.info/www/bejne/aYu6poOMjNZqppY Does that help?

[@kamran_desu]

Excellent explanation of cross validation and it's wonders - thanks for the improvement recommendations also

[@dataschool]

Thanks for your kind comment, and you're very welcome!

[@kamran_desu]

I actually used your method on my GBM model, something like a 10-fold stratified cross-validation on 80% of the data for hyperparameter tuning e.g. max depth, min rows, etc. in a search grid, and then kept 20% as hold-out set, works quite consistently :)

[@dataschool]

Great to hear!

[@Kevin7896bn]

One of the best explanation. Thanks

[@dataschool]

Thanks for your kind words!

[@hppeng]

THANK YOU SO MUCH. Wonderful video series. Well done. Thanks again.

[@dataschool]

You're very welcome! I'm glad the series is helpful to you!

[@rabellomusic]

you are amazing. Thank you for creating this course.

[@dataschool]

You're very welcome! I'm glad it's helpful to you!

[@sarabroadcasting9591]

This is really a very good video. Easy to understand

[@dataschool]

Thanks!

[@user-vt3uh2in7o]

best explanation, easy to understand. thank you so much

[@dataschool]

Thank you!

[@riderblack6401]

I will always be your audience. Your teaching saves me. lol

[@dataschool]

Glad to hear! :)

[@manalelai2598]

What a great material ! Thanks a million

[@dataschool]

You are very welcome!

[@swagatmishra9350]

Thank you very much for such a very beautiful explanation!!!

[@dataschool]

🙏

[@DTPwr]

All about cross validation in one video , THAAAAAAAAAAAAAAAAAAAAAANK YOU

[@dataschool]

You're very welcome! :)

[@abmsaroar2829]

Very Informative and well presented. Thanks a lot for sharing

[@dataschool]

You're very welcome! Hope you enjoy the rest of the series: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@brendensong8000]

Another great class!!!!

[@dataschool]

Thank you!

[@abhisheksaxena200]

Just awesome.. beautifully explained.. . Thanks a ton

[@dataschool]

You're very welcome!

[@libardomm.trasimaco]

Very clear and useful video. Thanks you so much

[@dataschool]

You're welcome!

[@NguyenDuy-jd6sm]

OMG those video are the bomb, Nice job mate !

[@dataschool]

Thank you!

[@matinafragkogianni1376]

Great video, thanks a lot!

[@dataschool]

+Matina Fragkogianni You're very welcome, I'm happy to help!

[@rusmat0173]

Absolutely brilliant! Thank you.

[@dataschool]

You're welcome!

[@sevicore]

Good video. Maybe is for my level of english but i dont understand why at 24:00 we compare the accuracy of KNN with Linear Regression when KNN is used for classification and Linear Regression is used for regression. I know Cross Validation works for both but the response variable in both cases should be different, since for KNN should be categorical and for LR should be continuos. Great series, enjoying them so far. Thanks for the good content :)

[@dataschool]

I'm comparing KNN with Logistic Regression, which is used for classification. Hope that helps!

[@patite3103]

Awesome video! Thank you a lot!

[@dataschool]

You're welcome!

[@shivanishrivastava4968]

Thank you so much, Sir!! Amazing tutorials :)

[@dataschool]

You're very welcome! :)

[@comeinwiththerain19]

This was super helpful, thank you.

[@dataschool]

Great to hear!

[@charlotteramos2267]

Hi Kevin, how are you!! First of all, thanks for your videos, you are awesome for posting them for us, and also because you are a great teacher and very good explaining. I have two doubts: 1. why is cross validation better than fitting a model (or training it) with all the data? 2. is cross validation a usefull method for timeseries?

[@reshaknarayan3944]

Saved my day! May god bless you.

[@dataschool]

Thanks!

[@davidavornyo128]

This is absolutely excellent 👍

[@dataschool]

Thank you so much! 🙏

[@javonnii436]

Great video! The only line of code that I needed to update is reshaping the data to pass into the binarize function and then flatten the return ndarray. '''y_pred_class_2 = binarize(y_pred_prob.reshape((192,1)), threshold=0.3).flatten() '''

[@Dexter01]

You are charismatic, thank you!

[@dataschool]

Thanks!

[@jongcheulkim7284]

Thank you so much. This is very helpful^^

[@dataschool]

Glad it was helpful!

[@jaimegonzalezsuarez1566]

good work you explain everything very well

[@dataschool]

Thanks so much for your comment!

[@mikemyers9524]

wow...very CLEAR. very HELPFUL

[@dataschool]

Thanks!

[@sridhaarrb]

Good Vedio, it make concepts clear and great to understand...

[@dataschool]

Thanks!

[@NipunRawat08]

The best tutorials ever!!

[@dataschool]

Thanks!