How to find the best model parameters in scikit-learn

In this video, you'll learn how to efficiently search for the optimal tuning parameters (or "hyperparameters") for your machine learning model in order to maximize its performance. I'll start by demonstrating an exhaustive "grid search" process using scikit-learn's GridSearchCV class, and then I'll compare it with RandomizedSearchCV, which can often achieve similar results in far less time.
Download the notebook: github.com/jus...
Grid search user guide: scikit-learn.or...
GridSearchCV documentation: scikit-learn.or...
RandomizedSearchCV documentation: scikit-learn.or...
Comparing randomized search and grid search: scikit-learn.or...
Randomized search video: • Scikit Learn Workshop ...
Randomized search notebook: github.com/amu...
Random Search for Hyper-Parameter Optimization: www.jmlr.org/pa...
WANT TO GET BETTER AT MACHINE LEARNING? HERE ARE YOUR NEXT STEPS:
1) WATCH my scikit-learn video series:
• Machine learning in Py...
2) SUBSCRIBE for more videos:
www.youtube.co...
3) JOIN "Data School Insiders" to access bonus content:
/ dataschool
4) ENROLL in my Machine Learning course:
www.dataschool...
5) LET'S CONNECT!
- Newsletter: www.dataschool...
- Twitter: / justmarkham
- Facebook: / datascienceschool
- LinkedIn: / justmarkham

[@dataschool]

Having problems with the code? I just finished updating the notebooks to use *scikit-learn 0.23* and *Python 3.9* 🎉! You can download the updated notebooks here: github.com/justmarkham/scikit-learn-videos

[@compton8301]

Ans thanks for updating it.

[@MrGuruPuru]

just brilliant. Not an inch of fluff in what you teach and you do it so beautifully.

[@dataschool]

Wow, thank you so much for your very kind words!

[@preethamreddy6726]

@@dataschool Thank you for such a nice explanation. Can I know this please why I can't get the first element of the tuple? Thank you print (grid.cv_results_[0]) KeyError Traceback (most recent call last) in ----> 1 print (grid.cv_results_[0]) KeyError: 0

[@anefuoche1053]

At some point my eyes became teary while watching this series. I have never come across such an amazing and passionate teacher. you explain every single thing, even the questions that pop up in my mind it feels like you foresee and address them, what's baffling is that you even tag them as questions before answering them. The additional resources are also pure gold. My God will bless you sir, may you live long and always be happy

[@mamacita5636]

Pls don’t stop making these videos you’re saving my dissertation 😭😭 thank you so much

[@dataschool]

You're very welcome!

[@roshanrajsingh4838]

This is straight tutorial I've ever seen. I've not skipped a second of this video.

[@dataschool]

Glad it's helpful to you! 🙌

[@kevinalkindy]

Thank you for making this video. What a crystal clear explanation!

[@dataschool]

You're very welcome! 🙏

[@jatingogia4633]

I like the way you explain everything so slowly and briefly. Thank you for such a quality content on ML!

[@dataschool]

Thanks for your kind words!

[@RS-el7iu]

its great i came across your clips... one of the BEST explantions out there. thanks a lot

[@dataschool]

You're welcome!

[@perevales]

Your videos are excellent. You have had the ability of explaining ML using Python in a very accessible way. I usually use R but now thanks to your great tutorials I will start using Python. Thanks and please don't stop producing these fantastic tutorials.

[@dataschool]

+Pedro Carmona Ibáñez Thanks for your kind comments! I just published a new tutorial (55 minutes): kzbin.info/www/bejne/bmbHpZyDr56nidE As well, I'm teaching a new course on Machine Learning with Text: www.dataschool.io/learn/

[@dhazra1]

Kevin, you are an awesome teacher. Not only your knowledge; your teaching style is also so good that it can make extremely happy both the new comers into machine learning as well as those are already in this field but need to understand the concepts further. I have gone through quite a few of your videos in youtube and everywhere it's just outstanding !!!!

[@dataschool]

Wow, thank you very much for your kind comments! I'm so glad the videos have been helpful to you!

[@shahriarrahman6482]

I watched the whole playlist in 1 day !! This is one of if not the best sklearn tutorials out..

[@dataschool]

Awesome, thank you!

[@pankajmathur1504]

Just want to say, all the video in Scikit series are excellent. Especially you have made so easy to understand the complex topic of cross validation and optimal parameter tuning. Keep up the good work. Can't wait for the next one...

[@dataschool]

+Pankaj Mathur Awesome! Thanks so much for your kind words.

[@vl4n7684zt]

If you are running Python 3.4 and getting a "Parameter values should be a list" error, then make the following adjustment to the code to properly read in the range: k_range = list(range(1,31)) Also, as always, print statements in 3.4 require parentheses.

[@dataschool]

+Stacy H Thanks for passing along that tip!

[@Dualphase90]

Thanks Stacy! :)

[@chetankv7218]

We can use numpy.arange(1,31) instead of range.

[@dataschool]

Thanks! I recently updated the code to use Python 3.6 and scikit-learn 0.19.1. The updated code can be found here: github.com/justmarkham/scikit-learn-videos

[@alexisparenty9445]

Whaoo! Brillant. Very well explained, you speak very clearly too. You certainly have a gift for teaching. Continue the good job

[@dataschool]

Thanks very much for your kind words! :)

[@nehathakar5622]

Your explanation makes each and every topic so simple and easy to understand. Really, I would like to thank you for your immense efforts for all your videos which are so informative and added resources help us to dig deeper in the related topics.

[@dataschool]

Thanks very much for your kind words! 😊

[@musasall5740]

You bring it in every video and make it easy to understand!

[@dataschool]

Thanks for your kind comment!

[@evanmiller29]

Data School thank you so much for these videos. You're showing all the topics that no one has put the time to explain to us plebs. Keep the good work up!

[@dataschool]

Evan Miller You're very welcome!

[@NB19273]

Succinct, self contained, and very clear. The best video on youtube ive found so far explaining parameter search. Thanks so much!

[@dataschool]

Excellent, that's great to hear!

[@BobDuCharme]

These videos are all really great, and I particularly like how the iPython notebooks serve so well to help me review what was in any particular video. I'd like to put in a vote for one of the future videos to cover clustering.

[@dataschool]

Bob DuCharme Great to hear... I spend a lot of time crafting these notebooks! And, thanks for the feedback, I will certainly consider clustering.

[@sidk5919]

U are an amazing teacher...thank you so much.

[@dataschool]

You're welcome! Thanks for your kind words :)

[@syedmohdsohail3506]

Kevin and Data School, you have been angels to me. I learned a hell lot in Machine Learning, thank you so much.

[@dataschool]

You're very welcome! I'm glad the videos have been helpful to you!

[@sinanwannous]

Thank you so much, one of the best and clearest explanation I've ever came across!!

[@dataschool]

Great to hear!

[@ohserra]

A M A Z I N G ! In one day I've learned what I need to get into machine learning in python and scikit-learn. I've been using matlab for a while, and it's overwhelming the possibilities of scikit and the IPython notebook. Thank you again! great job ;) I hope to keep hearing from you!

[@dataschool]

+Diogo Gonçalves Wow! What a kind and thoughtful comment! I greatly appreciate it.

[@imshafay]

Your the beast mate. I wanted to go through Machine Learning basics Revision, and I'm quite amazed this is the best Video I have ever watched on KZbin, in fact best from all the reading platforms as well.

[@dataschool]

Awesome! Thank you so much for your kind words! If you want to support Data School, my Patreon campaign might interest you: www.patreon.com/dataschool/overview

[@slowcoding]

Your lectures are very clear and esay to understand. Excellent!!!

[@dataschool]

Thanks for your kind comment!

[@betulchamplin3642]

your classes are simply AMAZING, thank you so much for all your efforts putting them together!

[@dataschool]

Wow, thank you!

[@dataschool]

*Note:* This video was recorded using Python 2.7 and scikit-learn 0.16. Recently, I updated the code to use Python 3.6 and scikit-learn 0.19.1. You can download the updated code here: github.com/justmarkham/scikit-learn-videos

[@bandhammanikanta1664]

This helped. Thank you :) :)

[@AxlRulz666]

I like your talking speed. It gives me enough time to absorb the concepts. May be its just me. Anyways, Thank you very much !

[@dataschool]

Great to hear - you're very welcome! :)

[@abhisheksalvi2438]

This is really helpful. Grateful for your efforts taken in creating this resource.

[@dataschool]

You're welcome!

[@minchin8041]

Best gridsearch tutorial on youtube. Thanks!

[@dataschool]

You're welcome! Glad it was helpful to you!

[@russelllavery2281]

Fantastic. We should all join his support group. 5 a month is cheap

[@mdinesk]

I would love to see more such videos, especially on handling huge datasets with categorical values

[@dataschool]

+Dinesh Kumar Murali Thanks for the suggestion! I'll take it into consideration.

[@behnoushpejhanmanesh4353]

Your teaching and explanations are amazing Kevin! Many thanks for all the effort you have put on preparing these tutorials. Do you have any tutorials about "DecisionTreeClassifier", to which you briefly refer in this video?

[@dataschool]

Thanks for your kind words! I don't have a video tutorial about decision trees, but I do have this lesson notebook: github.com/justmarkham/DAT8/blob/master/notebooks/17_decision_trees.ipynb Hope that helps!

[@compton8301]

You're amazing. Thanks for the free knowledge.

[@dataschool]

You're welcome!

[@brendensong8000]

Another amazing video! Thank you!

[@dataschool]

Thanks again!

[@fernandonogueira2291]

You´re the master! Thanks for such a well explained video.

[@dataschool]

Thank you!

[@pookiechips5496]

Thank you abundantly for everything that you are doing for the world.

[@dataschool]

You are very welcome! :)

[@nastarankianersi104]

Thank you so much for this clear and helpful tutorial and all the effort you put in to your work 🌸^^

[@dataschool]

You're very welcome!

[@LonglongFeng]

at 19:26, the knn.fit(X, y) is to display the classifier parameters only, since we already instantiate the knn model with the best parameters (1 step before the fitting). in another word, the 'fit' method will always give you the same output, no matter you fit(X_train, y_train), or fit(X_test, y_test), or fit(X, y). It just shows the best parameters in the model. just same as print(knn)

[@dataschool]

knn.fit(X, y) is a necessary step in order to train the model with the data. Instantiating the model in the previous step does not train the model, rather it just prepares the model to be trained using certain parameters.

[@cansu4333]

I have a question but I should express my compliments before asking it. The way you explain things is marvelous, and one can understand where things come from clearly. Really, thank you so much. The question is: What about splitting the data at the beginning of the whole process. And fit the grid search on x_train, y_train. And with the best parameter found, test the model on X_test. (I did these just to pretend to have real out-of-sample data). When I did it, the result was not that different. Now I wonder why that is so; Is it because of the smallness of the data or because what I did does not change the results so much anyway?

[@dataschool]

Thank you for your kind words! 🙏 As for your question: whether or not you need to split beforehand depends on your goals. It's complicated to explain briefly, I'm sorry! But I'll be covering it in my next ML course. Subscribe here for updates: www.dataschool.io/subscribe/

[@cansu4333]

@@dataschool thank you so much. ☺️

[@RexhepShijaku]

15:08 just to add something, there is another advise like this : it is better to select an odd value for k, since it may avoid the ties caused during classification . In this case [in the example on video] we have best scores when k=13, k=18, k=20, so probably it picked 13 because it is an odd number. I may be wrong but I heard it somewhere :)

[@DEEPAKKUMAR-sw6sb]

mesmerized by your explanations. Thank you so much!

[@dataschool]

You're very welcome!

[@ramlimbu886]

Pure gold! Thank you, Kevin!

[@dataschool]

You're very welcome! Glad this video was helpful to you!

[@lokeshpaladugula5793]

really awesome.... great work man... and thank youuuuu

[@dataschool]

Thanks!

[@yangbadminlog]

Isn't that using range will create a list?Why should we need to type list(range(1,31)) instead of range(1,31) ?

[@dataschool]

The range function creates a list in Python 2, but it does not in Python 3. By explicitly converting the output to a list, you guarantee that the code will work in both Python 2 and 3.

[@fablapp]

Hi Kevin, thank you so much for sharing this with us. enjoying any single minute of your videos. One question: receiving a warning message when running the .predict method on the dataset: "Passing 1d arrays as data is deprecated in i0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample." Not sure I am fully understanding what is about. Thanks again!

[@dataschool]

Glad you are enjoying the videos! Regarding the warning message, it's complicated to explain, but I discuss it in detail in this blog comment: www.dataschool.io/linear-regression-in-python/#comment-2521926219 Hope that helps!

[@asneogy]

hello Kevin - really great videos, love your clear and precise way of explaining concepts yet keeping them accessible. a request - there are not too many seaborn tutorial videos out there. could you consider making some? thanks and keep rocking.

[@augustinemalamsha9251]

u r a great teacher, optimazation, gridsearch, exhaustive search were a mist to me, nut now bcz of u, they kind of on my finger tips

[@dataschool]

Great!

[@greettheceo395]

Ahhh so awesome tutorial , struggling with this part for very long , now it got clear Thanks Regards

[@dataschool]

Great to hear!

[@GuRuGeorge03]

Thank you so much for this. This is literally gold

[@dataschool]

You're welcome!

[@JackSimpsonJBS]

Thank-you so much for these amazing tutorials, I've learned so much from them and now I use scikit-learn frequently. Was this the final video or did you have any idea how many you were planning to make in this series?

[@dataschool]

***** Excellent, glad to hear! I do have more videos planned, though I haven't yet decided how many more I will make. The next one will come out in a few weeks... stay tuned!

[@MrMmahesh007]

simply superb, you have amazing teaching skills. Great explanations and material suggested. I wanted to switch my career to MLearning and have been watching all your playlists. Once I complete all your videos, what is the best way to master?any suggestions.

[@dataschool]

Thanks so much for your kind words! My thoughts are here: www.dataschool.io/launch-your-data-science-career-with-python/

[@swagatmishra9350]

Thank you very much for such a very beautiful explanation!!!

[@dataschool]

Thanks for appreciating!

[@SunilKalmady]

Brilliant! You have excellent teaching skills. Please help me understand this better. Like with feature selection, whether finding optimal model (hyper)parameters should also be performed ideally within each cross-validation iteration? If i am not mistaken, you recommend using whole dataset (unsplit) for grid search in this video. Doesn't it constitute as (some form of) training from entire data? I would be happy if you can answer. Anyways, great stuff!!

[@dataschool]

+Sunil Kalmady Thanks for your kind words! You are correct that when searching for the optimal model, feature selection and feature engineering should *ideally* occur within each cross-validation fold. However, the question is always whether the added complexity is worth the increase in reliability of your performance estimates. This is a nice, short Q&A on that topic: stats.stackexchange.com/questions/92502/cross-validation-feature-information-outside-the-fold

[@SunilKalmady]

+Data School Thank you for your answer!!

[@josephkarpinski9586]

For those using Python 3.7, range(1, 31) may give you issues. If you print k_range and see range(1, 31) instead of a list of numbers, Import numpy as np and replace range(1, 31) with np.arange(1, 31).

[@dataschool]

Thanks! You can also just wrap range in list: list(range(1, 31)) You can find my code (which has been updated for Python 3) here: github.com/justmarkham/scikit-learn-videos

[@001JaNe100]

For those, like me, who have an issue with the round line I suggest : best_scores.append(round(float(rand.best_score_), 3)) # Maybe due to Python 3.5 ## And of course "Merci Kevin"!

[@dataschool]

+Nebil Jabari That's interesting... rand.best_score_ should already be a float. What error do you get with the original line of code?

[@001JaNe100]

Any error message show up. It simply produce the array with all the numbers unrounded. 0.97999999999999998 insteed 0.98 No matter the value of the second argument (3, 4, 2 or 1...)

[@AbhisarMohapatra]

There's a humbleness in the way you teach and deliver the explanations. It helps me understand very very well whatever the topic is.

[@dataschool]

Wow, thank you! I really appreciate your comment.

[@MattyTkernow]

That is exactly what I was thinking. If you can't explain it simply, you don't understand it well enough (Einstein). And you sir know what you are taking about!

[@pradyparyal]

I like your way of explanation. Thank you so much for your effort. Could you please upload the videos on unsupervised learning.

[@dataschool]

Thanks for your kind words! I'll consider your suggestion for the future.

[@citizenR1203]

Great ! Thanks for your ipython notebook.

[@dataschool]

You're welcome!

[@aoife1902]

Absolutely brilliant, thank you so much!

[@dataschool]

You're very welcome!

[@uniqueraj518]

NIce video, 1. please correct me if i am wrong , with the concept of gridSearch is to select the best parameter and cross - validation is to for generalizing model so that overfitting does not occur. 2. If u train the SVM model using scikit learn library with the dataset other than iris, training time is really high as compared to other model, and performance is also poor, i am confused what can be the solution. 3. There the way in IPython that we can use multiple cllient at the same time(Parallel computing), but i am lacking idea how it can be implemented here , i mead in instantiating the model, fitting the model or predicting stage. i would be happy for getting answer. I am waiting for your next video, about unsuvervised learning (clustering)

[@dataschool]

+unique raj 1. Grid search is for parameter tuning, and cross-validation is for estimating how well your model will generalize. 2. Every model takes a different amount of time to train (depending on the model itself, the data, and the tuning parameters), and no particular model is guaranteed to work for any particular problem.

[@DJH3891]

very good explanation! I still got an issue in understanding your statement at 19:35 - I thought training the model not enirely on all the data available was key, because otherwise you would have no test-set to evaluate or overfit the model??

[@smaxwell89]

Hi Kevin, thank you for making these videos as they have served as a very informative introduction to machine learning for me. I've watched each video of the series and studied the code thoroughly. Any recommendations as to what to do next as far as teaching myself machine learning? I'm planning to go back through the videos and study all of the supplemental resources you've provided and then complete the "An Introduction to Statistical Learning with Applications in R" course. Is this a good plan of action or should I focus more so on Python and scikit-learn? My main goal is to use my Physics background along with what I've been teaching myself to become a Data Scientist, but I would greatly appreciate your opinion on the matter. Thank you.

[@dataschool]

+smaxwell89 That sounds like a great plan! Here's some advice that I give my data science students: github.com/justmarkham/DAT7/blob/master/other/advice.md

[@eldert1735]

Thanks for the video. I have a question. When you put the parameters in the GridSearchCV, you put the knn as the estimator. 'knn' was instantiated with 'n_neighbors = k', looking at the above code. Does GridSearchCV ignore the things inside the parenthesis of the estimator? Or do we have to put 'n_neighbors = k' inside the KNeighborsClassifier, instead of a specific number to make the code work?

[@dataschool]

The parameter grid passed to GridSearchCV will override the parameters passed to knn during instantiation. Hope that helps!

[@chetakabra8]

one of the best explanation using pathon notebook ..can you please upload video for feature selection

[@dataschool]

+akash kabra I'll consider making one for the future, thanks!

[@dataschool]

You might be interested in my recent video about feature selection: kzbin.info/www/bejne/j5Kufph3oa2ap7M

[@nathankong8732]

Hi, quick question, why don’t you split the data before grid search? Also how would you do a confusion matrix using this model?

[@hmscfch.7216]

Hi, Kevin. I noticed that the codes are running slower on my PC (which is a standard one). Can I know what requirements are you recommended on the hardware? Or, any solutions to improve the processing speed? Thank you. And thank you for your 'quick-understanding' tutorials.

[@dataschool]

I'm not sure what hardware recommendations to make, I'm sorry!

[@sandeeps_]

I found the video series very useful! Thank you! :) Do you plan to have more videos in the future?

[@dataschool]

***** You're welcome! I will be creating more videos in the series, though it will be a few weeks from now before I have time to make the next one.

[@marathiManus10]

Love simplicity in your approach! Excuse me for my naive question - So How to determine if cluster size 13 is optimal or 17 is optimal? If one was to use said KNN logic in a TRULY real life situation, should he choose 13 or 17?

[@dariuszspiewak5624]

Not sure if I can take on Kevin's role here but I remember from his teachings that you should choose the parameter that makes your model SIMPLER. Since 17 gives you simpler decision boundaries, you should choose 17. In general, in KNN you should use the highest parameter out of the best performing ones.

[@balkiprasanna1984]

Thank you so much for this video. You're just amazing.

[@dataschool]

Wow, thanks so much! I really appreciate it :)

[@AndreasMueller]

Very nice video series :) I'd be curious if you used any of my material apart from the video you linked to [everything I do is CC-0 unless published via a publisher so don't fear the lawyers]. I found the continuous talking / walking through for videos really hard, you do a much better job than me, I think.

[@dataschool]

+Andreas Mueller Thanks very much for the compliments - I'm flattered! No, I'm pretty sure I didn't use any outside materials, with the exception of some images. I basically created every notebook from scratch. Regarding the talking during videos: It takes some getting used to, for sure! I scripted this series (every word and every action), which made the recording easier, but took FOREVER (probably 150 hours of work for a 4-hour series!!) I changed tactics for my latest video series (pandas), which is mostly unscripted, and thus I can make videos a lot faster. Looking forward to your book! Tentative release date? :)

[@AndreasMueller]

+Data School Summer It was announced for June but I don't think we'll make that. Stay tuned for an early release. I scripted my O'Reilly series (tried without script first, totally failed) and it also to for ever. I'll check out your pandas videos. Keep up the good work! I'm glad this material is openly available -- mine is not :-/

[@dataschool]

+Andreas Mueller Cool, I'll be on the lookout for the book this summer!

[@mallikarjunsuram4913]

Brilliant work dude

[@dataschool]

Thanks!

[@marcelohanones9007]

Great explanation !!!!! Thank You !!! I have a question. Why that GridSearch returned 13 as the best param for n_neighbors while RandomizedGridSearch returned 17 (or 20 as in 08_grid_search.ipynb) ? Looking at the plot It seems one got the lowest possible value (13) and the other the highest(17 or 20).

[@dataschool]

I'm not sure I totally understand your question, sorry! But my general response would be that for such a small dataset, and on such an easy problem, the "best" result can vary slightly. Hope that is at least a bit helpful! :)

[@saudnaeem]

Learned a lot from this video. thank u

[@dataschool]

You're welcome!

[@nsm9dvarunbabu618]

sure definetly

[@suryagaur7440]

Brilliant as usual. Can you please make video on nested and non-nested Cross validation

[@dataschool]

Thanks for your suggestion!

[@deboratoshiekohara2723]

Thaanks! Congrats Its a great explanation! I was also checking some other videos of yours and a doubt appeared, I wanted to combine the pipe with the gridsearch. For instance I tried to put down the gridsearch within my pipeline and extract the results as pipe.cv_results_, however could not use the .cv_results with pipe. Could you give any hint about the combination pipe for preprocessing and then search grid? Maybe another topic for your videos Thanks!

[@giorgos253]

If I have to do a classification in text with 7-8 classes categorical , which is the best algorithm to use SVC or someone else? Can you give a hint ?

[@thisaintarf]

thankyu very much sir, this video helps me a lot

[@dataschool]

Great!

[@jrabyssdragon]

Hi Kevin! I've a quick question. In this video you explained the SearchCV methods for tuning parameters. Also, I've found that there are some functions to perform feature selection with scikit learn. But I was wondering if there's a way to perform both feature selection and hyperparameter tuning within the same pipeline in scikit-learn. Any insight is appreciated.

[@dataschool]

Sure, you can do a GridSearchCV of a Pipeline that contains both a feature selector and a model. This example from the documentation shows something similar: scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html Hope that helps!

[@aryanol975]

As far as I understand, the GridsearchCV uses all the data to find best k value. But why are we using the whole data while fitting knn? wouldn't it be overfitting?

[@dataschool]

The goal of finding the "best" K value is so that you can use that parameter value when fitting the model on all of the relevant training data. If you don't fit the model to all of the relevant training data, then you are throwing away valuable data that can be used to improve the model. Does that help?

[@ElectronicsInside]

Do I have to split the data using the train_test_split function in cross-validation technique ?? I have seen people using cross_val_score(SVC() ,x=x_train, y=y_train,cv=10) But in the video we do like cross_val_score(SVC() ,x, y,cv=10)

[@dataschool]

I teach the latter, but it depends on your model evaluation process. There is not one "right" evaluation process, but some are better than others. Hope that helps!

[@ElectronicsInside]

@@dataschool I also want to know that we have to scale our data before train/test split or after train/test split.Some people told me that you should scale your data after train/test split

[@dataschool]

After

[@iftaknayeem9863]

Say I let the best estimator clf = grid.best_estimator_ after fitting that grid to the entire dataset X and y. Would I be able to fit the estimator again on a subset of X and y as such: clf.fit(x_subset,y_subset) ..Or would that cause overfitting since that estimator already saw the entire X and y. Basically what I'm trying to ask is: is grid.best_estimator just the estimator itself or a model that has already seen data?

[@dataschool]

I'm pretty sure that "grid.best_estimator_" is just the estimator itself, whereas "grid" actually contains the fitted model. Hope that helps!

[@parijatbandyopadhyay8057]

The value for the scoring parameter is "neg_mean_squared" for KNeighborsRegressor but it's giving an error as "not a valid scoring value". Am I doing something wrong?

[@dataschool]

Sorry, it's hard to say without seeing your code! But maybe this post will help? www.dataschool.io/how-to-update-your-scikit-learn-code-for-2018/#scoringparametersforlossfunctionshavebeenrenamed

[@shaileshshrestha2198]

great video. One comment though. for using the best parameter to make predictions, one doesn't need to hard coded as you did. one can simply use bestclf = grid.best_estimator_ then bestclf.fit(X,y). Cheers

[@dataschool]

+shailesh shrestha Thanks for your comment. You are correct - I included that line of code for teaching purposes. However, you can actually skip the fitting step, since the GridSearchCV automatically refits the best model using all of the data. You can see an example in cell 24 of my notebook: github.com/justmarkham/scikit-learn-videos/blob/master/08_grid_search.ipynb

[@liuchengyu5420]

Hello, In the video, we are tuning max_depth and min_samples_leaf which are for decision tree. Why do we use KNN int he video? Jason

[@dataschool]

They are just two separate examples.

[@boringpsychology9737]

great thanks

[@dataschool]

You're welcome!

[@sreemantokesh3999]

Your tutorials are so so great. Thank you for everything Kevin. I see the GridsearchCV and RandomizedSearch is kind of similar to Gradient Descent and Stochastic Descent. Am I wrong??? Can yo do a video where you explain Stochastic Gradient and it's Variation??

[@dataschool]

Thanks for your suggestion! I'll consider it for the future.

[@n7router]

I am confused what value does fitting the whole data into KNN will add. the K was already fixed during instantiation as well as the weight, what parameter might change when fit is called?

[@dataschool]

The point of re-fitting the model with all of the data is so that the model can learn from all of the relevant training data, and not just the artificial training set created by train/test split. No parameters will change when fit is called. Hope that helps!

[@LazzyFrog]

Can you please make a video on ensemble learning while using gridsearchCV ? And also use multiple algorithms and later ensemble to get the final result .

[@dataschool]

Thanks for your suggestion!

[@thatguywhohatesnoobs]

Does this method not take into account biases introduced by using the same dataset to tune hyperparameters and evaluate the model? How does this compare to nested cross-validation?

[@dataschool]

That's beyond the scope of what I can get into in a KZbin comment - thanks for the question though!

[@kirillpoznyak]

for Python 3 In[13] for mean, std, params in zip(grid.cv_results_['mean_test_score'], grid.cv_results_['std_test_score'], grid.cv_results_['params']): print("mean: " "%0.3f, " "std: " "%0.03f, " "params: " "%r" % (mean, std, params))

[@dataschool]

Thanks!

[@edisonliu4430]

Thank you for sharing the video. However, I was wondering what's difference of cross_val_score imported from sklearn.cross_validation and from sklearn.model_selection?

[@dataschool]

The sklearn modules were reorganized after I recorded this video, but they should be the same otherwise!

[@kishoredev6004]

may be its an update, it is now from sklearn.model_selection import GridSearchCV

[@MatthewAds]

Super helpful videos! What do you do when you want to do categorisation on a dataset which consists of many categorical variables? As I understand scikitlearn models expect numeric values always? What is the best way to convert datasets full of categorical data represented by strings into integer representations for use in scikitlearn models? Is that even an approach you would advise?

[@dataschool]

Matt Adshead Great question! Yes, scikit-learn expects numeric values. For unordered categorical features, you generally represent them as dummy variables. For ordered categorical variables, you can represent them using "sensible" numeric values. More details are available in part 3 of this notebook: nbviewer.ipython.org/github/justmarkham/DAT7/blob/master/notebooks/12_advanced_model_evaluation.ipynb

[@silasmurithi4706]

Awesome tutorial, I would like to know what happens if you tune parameters of different classification models to give similar predictions? Is it a good idea or a bad one? And how to go about it?

[@dataschool]

Glad you liked the tutorial! Not sure what you mean by "tuning to give similar predictions"... Instead, if you try tuning another model, your goal is to optimize for your chosen evaluation metric, not to make sure it gives similar predictions to any other model. Does that make sense?

[@kostasnikoloutsos5172]

I am wondering if it is worth to re-train your model with the predictions that have been made.This way, the model will have more train data but will it reduce the probability of model to make the right predictions ?

[@dataschool]

Generally speaking, you should incorporate new data into your model when possible. However, you wouldn't include your predictions without checking whether or not they are correct!

[@pinkanrout818]

simply Awesome!!!Thanks a lot for all your videos. I have a question here...Grid search we use to get the best hyper parameter as in this case n_neighbors. But how it will be helpful in linerar/logistic regression . In linear we are not using any tuning parameter (linreg.fit(x,Y)). In deep learning the role of learning rate comes may be that time it will be useful..not sure in scikit how it helps in linear regression.

[@dataschool]

You're right that linear regression has no tuning parameters. However, logistic regression does have a tuning parameter, so grid search is helpful there.

[@pengyan1906]

You are doing great!

[@dataschool]

Thank you!

[@thapajkamal]

How can I use Randomized search for MLPregressor? I could not find appropriate materials .. even example codes for that.

[@dataschool]

I'm sorry, I've never used MLPregressor and can't be of help. Good luck!