How do I encode categorical features using scikit-learn?

[@dataschool]

*Are you new to Machine Learning?* Watch my video series, "Introduction to Machine Learning in Python with scikit-learn": kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@arunjohn492]

Sir what about dummy variable trap , When we use Column Transformer ?

[@dataschool]

Great question! See this video: kzbin.info/www/bejne/hIrXqKysrtt3e80

[@GoredGored]

For beginners: When I tried to complete an ML project of say a simple model based on Logistic or Linear regression it used to take me about a month. As I was a beginner in Python, Pandas, SQL and the rest of it, I thought this will take me a long time to master and may be I am a late comer into this. But a year forward now and thanks to Data School, Sentdex, Krish naik, Statquest, Thinkful Webinar and more I am surprised that all I need is a day or less to complete these projects. Because of the meticulous analysis on Data School when I needed a deeper understanding that's where my gps leads me to. Thank you Data School.

[@dataschool]

You are so very welcome!

[@terryhenyo9216]

The Legendary Data Science guy is back!

[@dataschool]

Thank you for the warm welcome! 😄

[@altunbikubra]

Your guideline does not only involves basic codes, but it actually involves very practical and useful functions. I want to sincerely thank you for your effort!

[@dataschool]

Thanks very much for your kind words!

[@liquid_absabs1334]

There is something about your explanations, that i just get it instantly. You deserve an award

[@dataschool]

You are too kind, thank you!

[@dataschool]

Yes, that is the role of the OneHotEncoder.

[@fet1612]

00:58 1) It allows you to properly cross-validate a process rather than just a model. In other words, when you are doing cross-validation like cross_val_score, normally you just pass a model to it. Well, there are cases when that is not going to give you accurate results because you're doing the preprocessing outside of the cross-validation. So a pipeline, generally speaking, is useful because you can cross-validate a process that includes (a) *preprocessing* as well as (b) *model building*.

[@hieungotrung5411]

OMG!!! I’ve just started ML in kaggle for the past few weeks. Theres a lot of information to absorb but you teach us in the most understandable way and yet up-to-date question why we should use scikit instead of using dummies. This video is extremely helpful and informative. Thank you alot!!! Guess I gonna spend the rest of the day to watch all of your videos

[@dataschool]

Awesome! Glad to hear this was helpful to you 👍

[@420nyk]

Thanks, this helps a lot. Was scratching my head on pipeline and column transformer before this video. Also you got a very soothing voice and it helps to relax and really enjoy the learning.

[@dataschool]

Great to hear!

[@harshitarawat8941]

Man I love you. I just love you. I love your videos. I love the way you explain things. I love the pace of you videos. I love everything. Thank you.

[@dataschool]

Thank you so much, Harshita! 🙏

[@rommeltito123]

Dayyyyuuummmm.......why did I not stumble upon ur videos earlier ????!!!!!!

[@dataschool]

😄

[@harshalkulkarni511]

Preprocessing with pipeline was complex topic to understand for me before watching this video. Thanks a lot for the video.

[@dataschool]

You're very welcome! Glad it helped 👍

[@quocanhhbui8271]

My god I love your detailed solution. Even my 5yo sibling can understand it. Wonderful. Definitely worth a subscribe.

[@dataschool]

Awesome! 🙌

[@Rationalist-Forever]

I was looking for clear explanation of Pipeline for a long time. You nailed it. Crystal clear explanation and understood by watching one time. Thank you.

[@dataschool]

You're so very welcome! 🙏

[@chr1112]

you are the best tutor i have ever met , keep up the good work. Thank you

[@dataschool]

Wow, thanks!

[@christianiheanacho4976]

You are a high quality TEACHER , thank you very much.

[@dataschool]

You are very welcome! 😄

[@Tothefutureand]

Thx kevin, one of best & simplest explanations of pipeline

[@dataschool]

Glad it was helpful!

[@sandeep1026]

I feel fortunate that I stumbled across this video. Very well articulated. Slows down pace, so that folks can hear, understand and digest. Most videos I come across, seem to rush through the contet before one can digest. Thanks for taking time and sharing your knowledge

[@dataschool]

Thanks very much for your kind words! 🙏

[@Putinka1000]

Thank you for speaking slowly. It’s nice to listen to a non-English speaking person

[@dataschool]

You're very welcome! :)

[@Steven-se5jd]

just want to say thank you. I am a beginner and you teach much better than my professor.

[@dataschool]

Glad to hear I have been helpful! 🙏

[@tald747]

This is an excellent and simple explanation of this topic. I must say that you are a very talented in the way you teach! You choose your words in a way that emphasizes only the important and relevant staff. Thanks!!!

[@dataschool]

Wow, thank you!

[@amitsharma8337]

THANK YOU for this tutorial! Was wandering around the web to solve unexpected errors that came by following, apparently, outdated tutorials. If I have landed up on this tutorial the very first time, it would have saved me around 4 hours of useless surfing. Thanks again

[@dataschool]

That's awesome to hear... glad I could be of help! By the way, I'll be launching a full course covering these topics (and more)... sign up here to get notified when it launches: scikit-learn.tips

[@georgeognyanov]

God damn this video is good. I was struggling with column_transformer and pipelines till late last night. The options you suggest here are so much better and easier to understand for me. I am totally going through your "Introduction to Machine Learning in Python with scikit-learn" playlist soon. Thanks for putting this out!

[@dataschool]

You're very welcome! If you want to go deeper into this topic, you may want to check out my course: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@PaulBillingtonFW]

Thanks, for this clear and well paced tutorial.

[@dataschool]

Glad it was helpful!

[@horoshuhin]

thank you Kevin, very thorough explanation. I'm glad I found your channel. I like the way you teach.

[@dataschool]

Thank you so much! 🙏 That's great to hear!

[@krishkonnect814]

I just found solution to my problem after watching your video. Thanks a lot.

[@dataschool]

You're welcome!

[@fahadkhankhattak8339]

thank you so much!!!!! it was very helpful. yours is the only channel i come running to for help whenever im stuck somewhere. rich conent!! keep sharing these wonderful thingss

[@dataschool]

Thank you so much!

[@adarshr30]

After searching alot, i found this channel n i feel its best for me:)

[@dataschool]

Happy to hear that!

[@salonisamant5410]

Thank you for explaining the pipeline approach so well!

[@dataschool]

You're very welcome!

[@fet1612]

00:26 " What is the point of the pipeline? The point of the pipeline is to chain steps together sequentially. Normally, you put preprocessing steps and model building steps in a pipeline. Now, why should you build a pipeline? There are two main reasons."

[@luisguaniloquinones8936]

Thanks!

[@frankgiardina205]

Excellent! I was using the pandas dummies and your explanation of why pipeline and ohe is a better solution solves all the problems. thanks again

[@dataschool]

Glad it helped!

[@JainmiahSk]

Sir, just before 5 minutes I visited our channel to ask you the same question where it was difficult for me to encode multivariables in kaggles house prediction using advanced regression dataset. Fortunately and surprisingly you posted same. Thank you so much.

[@dataschool]

That's amazing! 🙌 I hope this video is helpful to you, and let me know if you have any questions!

[@JainmiahSk]

@@dataschool I have a problem with functions, I can't write custom functions in Python which is very important what to do sir?

[@dataschool]

@@JainmiahSk You can definitely write custom functions in Python!

[@aaqibsoomro5776]

You are a great teacher. Please make the tutorials or series for Data Visualization, In-Depth Data Analysis, and Cleaning, and Project Deployment, etc. Since after Learning Python and its libraries and ML, these are the next steps.

[@dataschool]

I have many more tutorials! Many of them are listed here: www.dataschool.io/launch-your-data-science-career-with-python/

[@nishantchaudhary7528]

That was really something amazingly explained, I was looking for all these topics to understand. I got it in one go. Thanks a ton.

[@dataschool]

You're very welcome!

[@jkore2554]

Thank you for this tutorial. I was working with logistic regression this week and was trying to figure out how to one hot encode for a categorical variable with hundreds of categories. I was getting 100% accuracy and precision so something wasn’t right. I’m going to try the steps that you outlined in this tutorial. Thanks.

[@dataschool]

Good luck!

[@dhananjaykansal8097]

Nice to have u back sir. This session was so fruitful. Thanks a ton. Keep it up!

[@dataschool]

That's awesome to hear!

[@TheAstralftw]

Finally someone explained me properly what is columns transformer and why we use pipeline. I would like you to put your course to udemy , then i ll buy it 100% .. maybe on average you will sell each course for less price, but trust me, you are explaining this so good, you can sell tens of thousands of courses in few months , ... or in the case you have this on udemy , please provide me with the link!

[@dataschool]

Thanks for your kind words and your suggestion! I know that many students like Udemy courses, but my values as a course creator don't align with their business model, and so I'm not currently interested in publishing a course there. I prefer to offer courses directly to interested students. Thanks for understanding!

[@jobihara]

Thankyou dataschool, it was not only helpful, it was great, enlightening and awesome.

[@dataschool]

What a nice thing to say, thank you so much! 🙏

[@aimenbaig6201]

i just discovered your channel and i gotta tell you , you got a permanent subscriber here!!! LOVE YOUR TEACHING STYLE!!!!!!!!!!!!!!!

[@dataschool]

Thank you! 🙏

[@sandeeppreetam]

Thank you good sir, this tutorial was better than many paid tutorials on Udemy. Blessed!

[@dataschool]

Glad it was helpful! 🙌

[@Takk6]

You are by far the best data science teacher on youtube. Can you make a video on creating your own custom transformers using it to modify your data, then using that custom transformer in a ColumnTransformer and a Pipeline?

[@dataschool]

Thanks for your suggestion! I'm working on a course that will likely cover that topic. Sign up here to get notified when it launches: scikit-learn.tips

[@lovejazzbass]

Kevin, it's 5:20am Winston-Salem time and I am digging this. I was very confused. Thank you so much.

[@dataschool]

Excellent!

[@David-fr7ee]

Great content, i am learning this in my college data science class. You did better than my professor!

[@CE-vd2px]

Are you undergrad or grad?

[@dataschool]

Thank you! 🙏

[@jatinshetty]

yo! Mind blown with the amount of things i learnt from this. Please keep at it!

[@dataschool]

Thank you! You might like my scikit-learn tips: github.com/justmarkham/scikit-learn-tips

[@Anarchy977]

Fantastic tutorial! Great teacher, best Machine Learning teacher on youtube! Thank you!

[@dataschool]

Thanks so much!

[@amitblizer4567]

Very clearly explained and helpful video - Thank you!

[@dataschool]

Glad it was helpful!

[@asimssheikh]

Impressive explanation, and logical approach to material presentation. You just got a new sub.

[@dataschool]

Welcome aboard!

[@abdelkaderkaouane1944]

Your explanation is very clear, thank you very much

[@dataschool]

You're welcome!

[@artyb3115]

Absolutely perfect and useful lessons! Thinking of becoming a patron member as I get a little more confident with ML

[@dataschool]

That would be awesome, thank you so much! You can join here: www.patreon.com/dataschool

[@brandonbermudez9047]

Absolute goat bruh, really thankful for your content

[@dataschool]

Thank you!

[@sanaullahkhanhassanzai8432]

Thank you very much and welcome back after a long time. You are as good as gets when it comes to Machine Learning. You have made me learn a lot. I cant wait for videos on deep learning. I hope you ll come up with deep learning soon. Thanks again

[@dataschool]

Thanks very much for your kind words, and for your suggestion as well!

[@NoWhiteGullibility]

Perfect timing, was just searching on pipelines the other day. Would be great to follow-up by tacking on Gridsearch in this context.

[@dataschool]

That's awesome to hear! I will definitely cover grid search of a pipeline at some point - thanks for the suggestion!

[@sowash2020]

You just gained another subscriber...this was super useful

[@dataschool]

Great to hear!

[@xinchenzou4558]

Thank you sir! You've really saved my life...

[@dataschool]

🙌

[@12345shipreck]

You are 100x better than my ML course teacher at uni. GG bro.

[@dataschool]

Thank you! 😄

[@sophiar5280]

Always love your step by step, clear lessons. Keep it coming.

[@dataschool]

Thank you!

[@gardnmi]

Since pandas get_dummies ignores non categorical values I've always done below but I might have to start using pipelines. Great video! train = pd.get_dummies(train) test = pd.get_dummies(test) test = test.reindex(columns=train.columns, fill_value=0)

[@dataschool]

Thanks for sharing! It's still okay to use get_dummies, but you may end up with a gigantic DataFrame that includes columns you're not interested in. Plus, you will definitely have problems if any of the categorical features in your test data include different values than your training data. Anyway, glad you liked the video and I hope to bring you over to Pipeline! 😉

[@gardnmi]

@@dataschool I ran into the misaligned shapes issues a lot. That's what test.reindex(columns=train.columns, fill_value=0) solved for me but it seems pipeline is a bit more elegant.

[@dataschool]

@@gardnmi Even though reindexing *appears* to fix the problem with misaligned shapes, there's a high likelihood that the columns of your test DataFrame no longer match the column ordering of your train DataFrame. That's a significant problem because it means that your features are in the wrong order in test, and thus your model will make incorrect predictions. Pipeline thankfully solves that problem!

[@gyanendergandhar]

Thanks alot for this tutorial Kevin. It really saved me😅

[@dataschool]

Glad to hear that!

[@eugenechew1476]

Why pay $900 at Uni when you can watch this amazing tutorial for free, and its wayyyy better!

[@dataschool]

Thanks! Stay tuned for a course that explores these topics is much more detail...

[@83vbond]

I paid $6000 :((

[@barulli87]

MIND BLOWN!!!! CV FOR A PROCESS!!! NOICE ONE!!

[@dataschool]

🤯

[@abdoulayebalde2139]

A very nice video that save my life I can see it is well explained keep uploading

[@dataschool]

Thanks!

[@joxa6119]

God this video answered my month unsolved question. God blessed you.

[@dataschool]

Great to hear!

[@Universe4mi]

Thanks, very clear and insightful!!

[@dataschool]

You're welcome!

[@ayyappahemanth7134]

Oh my god! after so much of exhaustive waiting another video came, which is far more useful than others for me! I just love your videos, the content was really useful in my real life, most of the youtube channels they just take the ideal ones which I might not encounter in my whole life! please do these videos regularly!

[@dataschool]

That is awesome to hear, thanks so much for your kind words! 🙏 Actually, I publish a new Q&A video every month for Data School Insiders at the $5 level: www.patreon.com/dataschool

[@salakkal]

Really great that you did a video like this . It just helped me a lot and I am really thankful for it brother . Keep going .

[@dataschool]

Thanks!

[@trentjones6468]

Amazing video. You are an excellent instructor. Got yourself a new subscriber :)

[@dataschool]

Thank you so much!

[@Susuwho]

this is so helpful that I have to comment. great job. thanks a lot

[@dataschool]

Glad it was helpful!

[@christianiheanacho4976]

I am enriched by this teaching.

[@dataschool]

Great to hear!

[@SaunakDey]

awesome explanation!! Thanks a lot

[@dataschool]

You're very welcome!

[@honprarules]

Amazing explanation, as always!

[@dataschool]

Thank you!

[@AjayVerma-xi2us]

Very good, it cleared my many doubts

[@dataschool]

Great to hear!

[@Pqj613]

It's a good tutorial for some reasons that you will explain later.:D

[@kishanlal676]

Thank you for this amazing video. Please do some videos on feature selection and scaling techniques in python!

[@dataschool]

I'm hoping to cover feature scaling in a future video, but I do have a video about feature selection: kzbin.info/www/bejne/j5Kufph3oa2ap7M Hope that helps!

[@eatbreathedatascience9593]

This video is excellent.

[@dataschool]

Thank you!

[@1stophchr]

thank you very much, very clear video

[@dataschool]

You're very welcome! 😄

[@hichamamchtkou7343]

Thank you very much, it 's very interesting and by the way, it is exactly what i need in my current ML project.

[@dataschool]

That's great to hear! Good luck with your project 🙌

[@hichamamchtkou7343]

@@dataschool thanks 👍

[@TheAdrianPardo]

Thank you so much! You're the best! Please go over scaling when you have a chance :) Question: Is is ok to leave in all of the OneHotEncoded columns with this pipe approach? I believe you previously mentioned how it's best to drop one of the columns to prevent multicollinearity. Any way to do this within the pipe?

[@dataschool]

You are so kind, thank you! 😊 Yes, I plan to cover StandardScaler at some point. Yes, it is okay to leave in all of the one-hot encoded columns. However, the "drop" parameter for OneHotEncoder (new in scikit-learn 0.21) does allow you to drop one feature per category. Hope that helps!

[@ramleo1461]

Even I had the same doubt... Thank you for clarifying 😊

[@absar66]

Great ! Great ! Great! tutorial..many thanks Kevin

[@dataschool]

You're very welcome!

[@surfzion]

Extremely helpful, thank you so much !!!

[@dataschool]

Glad it helped!

[@gisleberge4363]

Great example, educational.

[@dataschool]

Thank you!

[@nowhere5111]

This video helps a lot👍👍👍

[@dataschool]

Great!

[@victor-os9wq]

Thanks for such a detailed tutorial. I am working on a similar problem where I have multiple categorical features. In my dataset, the categorical variables has more than 90 possible values, as a result I am having an additional 121 columns when i use the Get.dummy, but I actually want just four levels. Please kindly advise me.

[@patrickmullan8356]

When applying the 'make_column_transfromer()' at 17:45 it returns the results (e.g., columns) in different order than the input data. Is there a way of making it return the columns in the same order. Or at least knowing which new columns belong to which original category - without having to do the math oneself? Especially if not using the introduced pipeline functionality, but relying on this transfromation-tool anyways, for different works for example, this seems to me to be a bit difficult in handling, or at least inspecting. Great introduction to the modules, anyways ;)

[@dataschool]

Great question! The ordering is actually predictable: it's the ordering of the columns that I specified to the ColumnTransformer (2 columns for Sex and 3 columns for Embarked), followed by the columns that I passed through (1 column for Pclass). Does that make sense?

[@patrickmullan8356]

@@dataschool Yes, makes sense. That's what i meant with "having to do the math" ... ;)

[@ramleo1461]

Hi, this will be very helpful.. Thank you for making this video!!

[@dataschool]

You are very welcome! 🙌

[@garychen6367]

Hi Kevin, thanks for the terrific tutorial. I have two questions about the feature processing, 1) when do we need to standardize or normalize the value features before training? I know that standardizing or normalizing the value features can affect the performance of some ML-algorithm, whether we should do it seems depends on what kind of ML-algorithm we adopted (i.e., it is better to standardizing value features when using ANN, but may not when using DT-related algorithm). 2) if we do need to standardize the value features, should we do it before encoding the categorical features or after? (I used to do a stupid way: first split the value and categorical features, then standardize the former ones and encode the latter ones, then concatenate them, is there a better way to do it?). Again, thank you for this fabulous tutorial.

[@siddhantmittal1157]

1)We usually standardize our data when we see that there is a huge difference in the values of different columns of our dataset. Let us consider an example of predicting the salaries of employees in a firm. Different attributes can include its year of Exp., his age and salary as our target variable. Our age and YOE column can have values from 20-60 and 1-15 but our salary can have values such as 30000, 50000 and like that. this can affect our model and also affect the error as when these values will be fitted in our algorithm, then salary column will have more weightage(if not standardize) therefore we need to convert the data 2) we do that after converting categorical data into numbers. Thank you PS : Correct me if I am wrong.

[@garychen6367]

​@@siddhantmittal1157 Hi Siddhant, thank you very much for answering, it really helps. For the second answer, does that mean that we need to first encode the categorical features (e.g, after encoding the categorical features part would be binary numbers like ( [[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]) , then scale it? Thus after scaling the above categorical features would be changed from the binary number structure into a dataset with different numbers, e.g., [[ 1.414, -0.707, -0.707], [-0.707, 1.414, -0.707], [-0.707, -0.707, 1.414]]), does that matter? Thank you for your reply again! Updated：Hi I think I found the answer for my question 2), there is an example in scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_0_23_0.html#sphx-glr-auto-examples-release-highlights-plot-release-highlights-0-23-0-py which indicates we should encode the categorical features and scale value features separately, then concatenate/combine them together for training!

[@MohammadrezaMokhtari-qh2yg]

amazing information. wow! thank you so much man.

[@dataschool]

You're very welcome!

[@pivotai525]

Simply the best!!

[@dataschool]

Thank you!

[@WafazAli-b4u]

Very Well Explained..

[@dataschool]

Thank you!

[@schuylerblasy2192]

This is a really interesting video. Column_transformer is sort of like a pipeline in itself. Kind of reminds me of vectotassembler in Spark/Pyspark.

[@dataschool]

Thanks Sky! One important difference is that ColumnTransformer stacks results side-by-side, whereas Pipeline feeds the output of one step to the input of the next step.

[@brendensong8000]

I love it! Amazing tips!

[@dataschool]

Thank you!

[@cogcog312]

Just excellent. Thanks! I am very new to data science so please bear with me. Question - "For a dataset that has several categorical features each column with a lot of different values (say each categorical column has 100 different values as opposed to just 2 for Gender - male or female), after using onehotencoder to convert them to unordered numerical values, the number of table columns increases astronomically. Then you run the model and say one or more of the categorical features are amongst the most useful, how do you reverse or convert back these encoded features to know which categorical feature each represents?

[@dataschool]

I'm not sure off-hand, sorry!

[@sihle_za]

Simply the best.

[@dataschool]

Thank you!

[@sonalisingh2136]

Just AweSomE

[@dataschool]

Thank you!

[@vincecarter7500]

thanks a lot for helping everyone out, was just wondering if you will be uploading more videos in the future

[@dataschool]

Yes! I just started posting again last week. Thanks for watching!

[@salseid1033]

Your tutorial is informative as always. May you prepare a tutorial how to interprete model. Like 'Black Box' interpretation in RF. Thank you.

[@dataschool]

Thanks for your suggestion! I'll consider it for the future!

[@oeb5542]

Just another amazing video. 😄

[@dataschool]

Thank you so much for your kind words! 😊

[@Narriz]

This is amazing.

[@dataschool]

Thank you! You might be interested in this course: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@KVishya]

Hi Kevin, thank you so much for the wonderful explanation, could you also explain how to use GridSearch or RandomizedSearch along with Pipelines?

[@dataschool]

Great suggestion! I'm working on a tutorial that will be published on KZbin in late April. It will include that topic. Stay tuned!

[@adityakharwade9501]

Awesome video and thank you for this explanation!!! I have one request could you please make video on PCA

[@dataschool]

Thanks for your suggestion!

[@zohrehvahdati787]

Thank you so much.😍😍🙏🙏👍👍 It helped me a lot.

[@dataschool]

Great to hear!

[@amitkumards5609]

No doubt video is great, But one question, if I use Random Forest and want to know the feature importance with feature names(by using column transformers we will end up having an array without any column names, ex: after one hot encoding category name should be the column name, but that is not happening with this setup) how can we do it with this setup ?

[@dataschool]

Great question! Under certain conditions, you can use the ColumnTransformer's get_feature_names method to extract the feature names.