How do I encode categorical features using scikit-learn?

[@dataschool]

*Are you new to Machine Learning?* Watch my video series, "Introduction to Machine Learning in Python with scikit-learn": kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@arunjohn492]

Sir what about dummy variable trap , When we use Column Transformer ?

[@dataschool]

Great question! See this video: kzbin.info/www/bejne/hIrXqKysrtt3e80

[@GoredGored]

For beginners: When I tried to complete an ML project of say a simple model based on Logistic or Linear regression it used to take me about a month. As I was a beginner in Python, Pandas, SQL and the rest of it, I thought this will take me a long time to master and may be I am a late comer into this. But a year forward now and thanks to Data School, Sentdex, Krish naik, Statquest, Thinkful Webinar and more I am surprised that all I need is a day or less to complete these projects. Because of the meticulous analysis on Data School when I needed a deeper understanding that's where my gps leads me to. Thank you Data School.

[@dataschool]

You are so very welcome!

[@terryhenyo9216]

The Legendary Data Science guy is back!

[@dataschool]

Thank you for the warm welcome! 😄

[@liquid_absabs1334]

There is something about your explanations, that i just get it instantly. You deserve an award

[@dataschool]

You are too kind, thank you!

[@dataschool]

Yes, that is the role of the OneHotEncoder.

[@altunbikubra]

Your guideline does not only involves basic codes, but it actually involves very practical and useful functions. I want to sincerely thank you for your effort!

[@dataschool]

Thanks very much for your kind words!

[@hieungotrung5411]

OMG!!! I’ve just started ML in kaggle for the past few weeks. Theres a lot of information to absorb but you teach us in the most understandable way and yet up-to-date question why we should use scikit instead of using dummies. This video is extremely helpful and informative. Thank you alot!!! Guess I gonna spend the rest of the day to watch all of your videos

[@dataschool]

Awesome! Glad to hear this was helpful to you 👍

[@420nyk]

Thanks, this helps a lot. Was scratching my head on pipeline and column transformer before this video. Also you got a very soothing voice and it helps to relax and really enjoy the learning.

[@dataschool]

Great to hear!

[@Rationalist-Forever]

I was looking for clear explanation of Pipeline for a long time. You nailed it. Crystal clear explanation and understood by watching one time. Thank you.

[@dataschool]

You're so very welcome! 🙏

[@chr1112]

you are the best tutor i have ever met , keep up the good work. Thank you

[@dataschool]

Wow, thanks!

[@harshalkulkarni511]

Preprocessing with pipeline was complex topic to understand for me before watching this video. Thanks a lot for the video.

[@dataschool]

You're very welcome! Glad it helped 👍

[@harshitarawat8941]

Man I love you. I just love you. I love your videos. I love the way you explain things. I love the pace of you videos. I love everything. Thank you.

[@dataschool]

Thank you so much, Harshita! 🙏

[@rommeltito123]

Dayyyyuuummmm.......why did I not stumble upon ur videos earlier ????!!!!!!

[@dataschool]

😄

[@sandeep1026]

I feel fortunate that I stumbled across this video. Very well articulated. Slows down pace, so that folks can hear, understand and digest. Most videos I come across, seem to rush through the contet before one can digest. Thanks for taking time and sharing your knowledge

[@dataschool]

Thanks very much for your kind words! 🙏

[@tald747]

This is an excellent and simple explanation of this topic. I must say that you are a very talented in the way you teach! You choose your words in a way that emphasizes only the important and relevant staff. Thanks!!!

[@dataschool]

Wow, thank you!

[@Steven-se5jd]

just want to say thank you. I am a beginner and you teach much better than my professor.

[@dataschool]

Glad to hear I have been helpful! 🙏

[@quocanhhbui8271]

My god I love your detailed solution. Even my 5yo sibling can understand it. Wonderful. Definitely worth a subscribe.

[@dataschool]

Awesome! 🙌

[@Tothefutureand]

Thx kevin, one of best & simplest explanations of pipeline

[@dataschool]

Glad it was helpful!

[@fahadkhankhattak8339]

thank you so much!!!!! it was very helpful. yours is the only channel i come running to for help whenever im stuck somewhere. rich conent!! keep sharing these wonderful thingss

[@dataschool]

Thank you so much!

[@amitsharma8337]

THANK YOU for this tutorial! Was wandering around the web to solve unexpected errors that came by following, apparently, outdated tutorials. If I have landed up on this tutorial the very first time, it would have saved me around 4 hours of useless surfing. Thanks again

[@dataschool]

That's awesome to hear... glad I could be of help! By the way, I'll be launching a full course covering these topics (and more)... sign up here to get notified when it launches: scikit-learn.tips

[@christianiheanacho4976]

You are a high quality TEACHER , thank you very much.

[@dataschool]

You are very welcome! 😄

[@horoshuhin]

thank you Kevin, very thorough explanation. I'm glad I found your channel. I like the way you teach.

[@dataschool]

Thank you so much! 🙏 That's great to hear!

[@fet1612]

00:58 1) It allows you to properly cross-validate a process rather than just a model. In other words, when you are doing cross-validation like cross_val_score, normally you just pass a model to it. Well, there are cases when that is not going to give you accurate results because you're doing the preprocessing outside of the cross-validation. So a pipeline, generally speaking, is useful because you can cross-validate a process that includes (a) *preprocessing* as well as (b) *model building*.

[@PaulBillingtonFW]

Thanks, for this clear and well paced tutorial.

[@dataschool]

Glad it was helpful!

[@georgeognyanov]

God damn this video is good. I was struggling with column_transformer and pipelines till late last night. The options you suggest here are so much better and easier to understand for me. I am totally going through your "Introduction to Machine Learning in Python with scikit-learn" playlist soon. Thanks for putting this out!

[@dataschool]

You're very welcome! If you want to go deeper into this topic, you may want to check out my course: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@Putinka1000]

Thank you for speaking slowly. It’s nice to listen to a non-English speaking person

[@dataschool]

You're very welcome! :)

[@salonisamant5410]

Thank you for explaining the pipeline approach so well!

[@dataschool]

You're very welcome!

[@adarshr30]

After searching alot, i found this channel n i feel its best for me:)

[@dataschool]

Happy to hear that!

[@krishkonnect814]

I just found solution to my problem after watching your video. Thanks a lot.

[@dataschool]

You're welcome!

[@Takk6]

You are by far the best data science teacher on youtube. Can you make a video on creating your own custom transformers using it to modify your data, then using that custom transformer in a ColumnTransformer and a Pipeline?

[@dataschool]

Thanks for your suggestion! I'm working on a course that will likely cover that topic. Sign up here to get notified when it launches: scikit-learn.tips

[@aimenbaig6201]

i just discovered your channel and i gotta tell you , you got a permanent subscriber here!!! LOVE YOUR TEACHING STYLE!!!!!!!!!!!!!!!

[@dataschool]

Thank you! 🙏

[@lovejazzbass]

Kevin, it's 5:20am Winston-Salem time and I am digging this. I was very confused. Thank you so much.

[@dataschool]

Excellent!

[@nishantchaudhary7528]

That was really something amazingly explained, I was looking for all these topics to understand. I got it in one go. Thanks a ton.

[@dataschool]

You're very welcome!

[@jobihara]

Thankyou dataschool, it was not only helpful, it was great, enlightening and awesome.

[@dataschool]

What a nice thing to say, thank you so much! 🙏

[@dhananjaykansal8097]

Nice to have u back sir. This session was so fruitful. Thanks a ton. Keep it up!

[@dataschool]

That's awesome to hear!

[@aaqibsoomro5776]

You are a great teacher. Please make the tutorials or series for Data Visualization, In-Depth Data Analysis, and Cleaning, and Project Deployment, etc. Since after Learning Python and its libraries and ML, these are the next steps.

[@dataschool]

I have many more tutorials! Many of them are listed here: www.dataschool.io/launch-your-data-science-career-with-python/

[@David-fr7ee]

Great content, i am learning this in my college data science class. You did better than my professor!

[@CE-vd2px]

Are you undergrad or grad?

[@dataschool]

Thank you! 🙏

[@TheAstralftw]

Finally someone explained me properly what is columns transformer and why we use pipeline. I would like you to put your course to udemy , then i ll buy it 100% .. maybe on average you will sell each course for less price, but trust me, you are explaining this so good, you can sell tens of thousands of courses in few months , ... or in the case you have this on udemy , please provide me with the link!

[@dataschool]

Thanks for your kind words and your suggestion! I know that many students like Udemy courses, but my values as a course creator don't align with their business model, and so I'm not currently interested in publishing a course there. I prefer to offer courses directly to interested students. Thanks for understanding!

[@Anarchy977]

Fantastic tutorial! Great teacher, best Machine Learning teacher on youtube! Thank you!

[@dataschool]

Thanks so much!

[@artyb3115]

Absolutely perfect and useful lessons! Thinking of becoming a patron member as I get a little more confident with ML

[@dataschool]

That would be awesome, thank you so much! You can join here: www.patreon.com/dataschool

[@sandeeppreetam]

Thank you good sir, this tutorial was better than many paid tutorials on Udemy. Blessed!

[@dataschool]

Glad it was helpful! 🙌

[@jatinshetty]

yo! Mind blown with the amount of things i learnt from this. Please keep at it!

[@dataschool]

Thank you! You might like my scikit-learn tips: github.com/justmarkham/scikit-learn-tips

[@frankgiardina205]

Excellent! I was using the pandas dummies and your explanation of why pipeline and ohe is a better solution solves all the problems. thanks again

[@dataschool]

Glad it helped!

[@JainmiahSk]

Sir, just before 5 minutes I visited our channel to ask you the same question where it was difficult for me to encode multivariables in kaggles house prediction using advanced regression dataset. Fortunately and surprisingly you posted same. Thank you so much.

[@dataschool]

That's amazing! 🙌 I hope this video is helpful to you, and let me know if you have any questions!

[@JainmiahSk]

@@dataschool I have a problem with functions, I can't write custom functions in Python which is very important what to do sir?

[@dataschool]

@@JainmiahSk You can definitely write custom functions in Python!

[@amitblizer4567]

Very clearly explained and helpful video - Thank you!

[@dataschool]

Glad it was helpful!

[@brandonbermudez9047]

Absolute goat bruh, really thankful for your content

[@dataschool]

Thank you!

[@sowash2020]

You just gained another subscriber...this was super useful

[@dataschool]

Great to hear!

[@jkore2554]

Thank you for this tutorial. I was working with logistic regression this week and was trying to figure out how to one hot encode for a categorical variable with hundreds of categories. I was getting 100% accuracy and precision so something wasn’t right. I’m going to try the steps that you outlined in this tutorial. Thanks.

[@dataschool]

Good luck!

[@sanaullahkhanhassanzai8432]

Thank you very much and welcome back after a long time. You are as good as gets when it comes to Machine Learning. You have made me learn a lot. I cant wait for videos on deep learning. I hope you ll come up with deep learning soon. Thanks again

[@dataschool]

Thanks very much for your kind words, and for your suggestion as well!

[@asimssheikh]

Impressive explanation, and logical approach to material presentation. You just got a new sub.

[@dataschool]

Welcome aboard!

[@12345shipreck]

You are 100x better than my ML course teacher at uni. GG bro.

[@dataschool]

Thank you! 😄

[@abdelkaderkaouane1944]

Your explanation is very clear, thank you very much

[@dataschool]

You're welcome!

[@sophiar5280]

Always love your step by step, clear lessons. Keep it coming.

[@dataschool]

Thank you!

[@gyanendergandhar]

Thanks alot for this tutorial Kevin. It really saved me😅

[@dataschool]

Glad to hear that!

[@xinchenzou4558]

Thank you sir! You've really saved my life...

[@dataschool]

🙌

[@ayyappahemanth7134]

Oh my god! after so much of exhaustive waiting another video came, which is far more useful than others for me! I just love your videos, the content was really useful in my real life, most of the youtube channels they just take the ideal ones which I might not encounter in my whole life! please do these videos regularly!

[@dataschool]

That is awesome to hear, thanks so much for your kind words! 🙏 Actually, I publish a new Q&A video every month for Data School Insiders at the $5 level: www.patreon.com/dataschool

[@NoWhiteGullibility]

Perfect timing, was just searching on pipelines the other day. Would be great to follow-up by tacking on Gridsearch in this context.

[@dataschool]

That's awesome to hear! I will definitely cover grid search of a pipeline at some point - thanks for the suggestion!

[@salakkal]

Really great that you did a video like this . It just helped me a lot and I am really thankful for it brother . Keep going .

[@dataschool]

Thanks!

[@barulli87]

MIND BLOWN!!!! CV FOR A PROCESS!!! NOICE ONE!!

[@dataschool]

🤯

[@trentjones6468]

Amazing video. You are an excellent instructor. Got yourself a new subscriber :)

[@dataschool]

Thank you so much!

[@abdoulayebalde2139]

A very nice video that save my life I can see it is well explained keep uploading

[@dataschool]

Thanks!

[@Susuwho]

this is so helpful that I have to comment. great job. thanks a lot

[@dataschool]

Glad it was helpful!

[@Universe4mi]

Thanks, very clear and insightful!!

[@dataschool]

You're welcome!

[@wexwexexort]

OK, you've said that we are cross validating not the model but the pipeline at around 22:00. This might be useful in some other case, but what's the point of splitting first then applying one hot encoding? Result should be same if you do the one hot encoding first. Right? Am I missing something?

[@dataschool]

You will have different results if there are small categories and (by chance) some of them only appear in test but not in train.

[@honprarules]

Amazing explanation, as always!

[@dataschool]

Thank you!

[@christianiheanacho4976]

I am enriched by this teaching.

[@dataschool]

Great to hear!

[@SaunakDey]

awesome explanation!! Thanks a lot

[@dataschool]

You're very welcome!

[@salmantabatabai]

What happens if the new dataset has some more categories than the training data. Like beside from C, S and Q in training, it also has something called B. How this is going to be handled?

[@dataschool]

Great question! You just have to tell OneHotEncoder how to handle unknown categories. See this video for details: kzbin.info/www/bejne/mHKZnox5ZsaSe8k

[@patrickmullan8356]

When applying the 'make_column_transfromer()' at 17:45 it returns the results (e.g., columns) in different order than the input data. Is there a way of making it return the columns in the same order. Or at least knowing which new columns belong to which original category - without having to do the math oneself? Especially if not using the introduced pipeline functionality, but relying on this transfromation-tool anyways, for different works for example, this seems to me to be a bit difficult in handling, or at least inspecting. Great introduction to the modules, anyways ;)

[@dataschool]

Great question! The ordering is actually predictable: it's the ordering of the columns that I specified to the ColumnTransformer (2 columns for Sex and 3 columns for Embarked), followed by the columns that I passed through (1 column for Pclass). Does that make sense?

[@patrickmullan8356]

@@dataschool Yes, makes sense. That's what i meant with "having to do the math" ... ;)

[@21Gannu]

wondering how wound one combine gridsearch and standarization steps

[@dataschool]

Great question! Standardization would either be one of the transformers in the ColumnTransformer or one of the steps in the Pipeline. GridSearchCV can be used to grid search the entire Pipeline. Hope that helps!

[@will6403]

Do you still have to split the data for training and test when using pipeline?

[@dataschool]

You can use train/test split as your model evaluation procedure for a pipeline, or you can ues cross-validation, or you can use both (depending on your goals). Hope that helps!

[@ajaysaikiranpenumareddy9809]

How can we map label encoded data in training to new incoming data for predicting any class.

[@gardnmi]

Since pandas get_dummies ignores non categorical values I've always done below but I might have to start using pipelines. Great video! train = pd.get_dummies(train) test = pd.get_dummies(test) test = test.reindex(columns=train.columns, fill_value=0)

[@dataschool]

Thanks for sharing! It's still okay to use get_dummies, but you may end up with a gigantic DataFrame that includes columns you're not interested in. Plus, you will definitely have problems if any of the categorical features in your test data include different values than your training data. Anyway, glad you liked the video and I hope to bring you over to Pipeline! 😉

[@gardnmi]

@@dataschool I ran into the misaligned shapes issues a lot. That's what test.reindex(columns=train.columns, fill_value=0) solved for me but it seems pipeline is a bit more elegant.

[@dataschool]

@@gardnmi Even though reindexing *appears* to fix the problem with misaligned shapes, there's a high likelihood that the columns of your test DataFrame no longer match the column ordering of your train DataFrame. That's a significant problem because it means that your features are in the wrong order in test, and thus your model will make incorrect predictions. Pipeline thankfully solves that problem!

[@modhua4497]

Thanks Kevin, do you have any video example that shows how to incorporate a self defined function in pandas pipeline?

[@AjayVerma-xi2us]

Very good, it cleared my many doubts

[@dataschool]

Great to hear!

[@olamartins]

Please can you teach on OneHotEncoding in sklearn 0.24.1? categorical_features is deprecated in the lower version. I'm stuck

[@eatbreathedatascience9593]

This video is excellent.

[@dataschool]

Thank you!

[@12345shipreck]

My categorical variable is y and the method you did with the column transformer just doesnt work on it. It gives me error :(

[@roopeshroope5988]

Is this tutorial explains about how to prepare dataset for k fold cross validation to avoid data leakage?

[@victor-os9wq]

Thanks for such a detailed tutorial. I am working on a similar problem where I have multiple categorical features. In my dataset, the categorical variables has more than 90 possible values, as a result I am having an additional 121 columns when i use the Get.dummy, but I actually want just four levels. Please kindly advise me.

[@AmirKhan_KnowTech]

if one of the column is having numerical data and it is to be normalized then in 'remainder' how we can do that?

[@IgnitedMountain]

Hello, in the last example. How is the NAN values handled. Are they removed by one of the methods or do you have to remove them by yourself?

[@frosty2164]

is there a way to split the data into train and test and check the accuracy in the pipeline after the dummy creation. In short i want to add the train split code in the pipeline?

[@dataschool]

Yes, you can use train/test split with a pipeline (instead of cross-validation). You fit the pipeline on the training set, make predictions on the testing set, and then check the results. Hope that helps!

[@amitkumards5609]

No doubt video is great, But one question, if I use Random Forest and want to know the feature importance with feature names(by using column transformers we will end up having an array without any column names, ex: after one hot encoding category name should be the column name, but that is not happening with this setup) how can we do it with this setup ?

[@dataschool]

Great question! Under certain conditions, you can use the ColumnTransformer's get_feature_names method to extract the feature names.

[@ratnakarmaurya3066]

hey do we not need to drop of a column to avoid dummy trap using (columntransformer class)?

[@dataschool]

See this video: kzbin.info/www/bejne/hIrXqKysrtt3e80

[@Camila-fv9qj]

Why you used fit_transform for the all data

[@hichamamchtkou7343]

Thank you very much, it 's very interesting and by the way, it is exactly what i need in my current ML project.

[@dataschool]

That's great to hear! Good luck with your project 🙌

[@hichamamchtkou7343]

@@dataschool thanks 👍

[@1stophchr]

thank you very much, very clear video

[@dataschool]

You're very welcome! 😄

[@yoniziv]

How would you deal with ordinal categorical variables (e.g. Small-Medium-Large...) since they are categorical but obviously have clear "numerical" order. Would you just convert the names to integers or is there a more methodical way to do it?

[@dataschool]

Great question! You would use OrdinalEncoder. See tip #6 here for an example: github.com/justmarkham/scikit-learn-tips Also, watch out for the tip #6 video in this playlist, which I'll publish on October 29: kzbin.info/aero/PL5-da3qGB5ID7YYAqireYEew2mWVvgmj6 Hope that helps!

[@elvykamunyokomanunebo1441]

Hi, When we encode Embarked, we end up with 3 dummy variables, however, it's not specified if any of these new features is dropped so that it can be the reference feature. Why is that? Thanks in advance

[@dataschool]

My thoughts on dropping a feature are here: kzbin.info/www/bejne/hIrXqKysrtt3e80

[@riteshtripathi8626]

Hello teacher, please explain in layman's term about the final predicted values that you got (what are they): pipe.predict(X_new) Results: array([1, 0, 1, 1, 0])#how should I be interpreting these values. Thanks

[@kyledesana6441]

Those are the predicted values for whether the passengers in the X_new data frame survived or not (Survived is a dummy variable)

[@riteshtripathi8626]

@@kyledesana6441 Thank you, this clears the doubt. :)

[@dataschool]

Great question! That's the prediction of survived (1) or did not survive (0) for the five samples in the new data.

[@joxa6119]

God this video answered my month unsolved question. God blessed you.

[@dataschool]

Great to hear!

[@raymondlee7280]

After I used the OneHotEncoder, I could generate a number of columns attributing dummy variables, but I found it difficult to add to the original dataframe for other data-preprocessing steps, it shows 'ValueError: could not broadcast input array from shape (10,3) into shape (10)'. I know that there is a difference in dimension between the column being one-hot-encoded and the dummy variables columns. How could I replace the column with categorical variables with the new generated columns? or I just use ColumnTransformer to handle this issue? Thank you.

[@dataschool]

Great question! Ideally, you should be doing all of your preprocessing within scikit-learn. Here's why: kzbin.info/www/bejne/r6eXkpd6fMh5e5o

[@ktay895]

Am I missing something -- shouldn't there be a train-test-split somewhere?

[@dataschool]

Great question! Cross-validation is a model evaluation procedure, which is a superior alternative to train/test split for model evaluation. That being said, there is some utility in first doing a train/test split, then performing hyperparameter tuning (usually with a grid search) on the training set, and then testing the best model on the testing set. That gives you a more reliable estimate of out-of-sample performance, but (1) it adds code complexity, (2) it sacrifices some of your training data that could be used for hyperparameter tuning, and (3) it's entirely unnecessary if your only goal is to find the best parameters for your pipeline. In short: whether or not you should also include a train/test split is situation-dependent.

[@adityakharwade9501]

Awesome video and thank you for this explanation!!! I have one request could you please make video on PCA

[@dataschool]

Thanks for your suggestion!

[@TheAdrianPardo]

Thank you so much! You're the best! Please go over scaling when you have a chance :) Question: Is is ok to leave in all of the OneHotEncoded columns with this pipe approach? I believe you previously mentioned how it's best to drop one of the columns to prevent multicollinearity. Any way to do this within the pipe?

[@dataschool]

You are so kind, thank you! 😊 Yes, I plan to cover StandardScaler at some point. Yes, it is okay to leave in all of the one-hot encoded columns. However, the "drop" parameter for OneHotEncoder (new in scikit-learn 0.21) does allow you to drop one feature per category. Hope that helps!

[@ramleo1461]

Even I had the same doubt... Thank you for clarifying 😊

[@adamwathieu1878]

Is there a way to add a header row to the numPy Array so that you know what each hot encoded column represents?

[@dataschool]

Great question! You would just have to convert the NumPy array back to a pandas DataFrame in order to add column names, because NumPy arrays can't have a header row.

[@lakshmitejaswi7832]

At train u have 2 different values and at test time if you have 3 different values in a column how to do it?

[@dataschool]

Great question! See the handle_unknown parameter of OneHotEncoder: scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

[@messi3210]

Question: Why are we passing all the encoded features to the model? We should pass only k-1 dummy/encoded features when they have k categories. Or did you just not worry about it as you were trying to demonstrate the pipeline and column transform functionalities?

[@dataschool]

Great question! You can drop the first level, but the current "conventional wisdom" in the ML community is that you don't get any model performance benefit from dropping a level. Thus, if your goal is model performance, you should leave the first level intact just to minimize the complexity of your process.

[@surfzion]

Extremely helpful, thank you so much !!!

[@dataschool]

Glad it helped!

[@karinahulka1486]

so if I may ask - there's no point of creating one hot encoder for categorical or ordinal variables, which are numeric?

[@dataschool]

Great question! If they are already numeric, and they are logically ordered, then you are correct, there's no point in one-hot encoding them.