How do I encode categorical features using scikit-learn?

In order to include categorical features in your Machine Learning model, you have to encode them numerically using "dummy" or "one-hot" encoding. But how do you do this correctly using scikit-learn?
In this video, you'll learn how to use OneHotEncoder and ColumnTransformer to encode your categorical features and prepare your feature matrix in a single step. You'll also learn how to include this step within a Pipeline so that you can cross-validate your model and preprocessing steps simultaneously. Finally, you'll learn why you should use scikit-learn (rather than pandas) for preprocessing your dataset.
AGENDA:
0:00 Introduction
0:22 Why should you use a Pipeline?
2:30 Preview of the lesson
3:35 Loading and preparing a dataset
6:11 Cross-validating a simple model
10:00 Encoding categorical features with OneHotEncoder
15:01 Selecting columns for preprocessing with ColumnTransformer
19:00 Creating a two-step Pipeline
19:54 Cross-validating a Pipeline
21:44 Making predictions on new data
23:43 Recap of the lesson
24:50 Why should you use scikit-learn (rather than pandas) for preprocessing?
CODE FROM THIS VIDEO: github.com/justmarkham/scikit...
WANT TO JOIN MY NEXT LIVE WEBCAST? Become a member ($5/month):
/ dataschool
=== RELATED RESOURCES ===
OneHotEncoder documentation: scikit-learn.org/stable/modul...
ColumnTransformer documentation: scikit-learn.org/stable/modul...
Pipeline documentation: scikit-learn.org/stable/modul...
My video on cross-validation: • Selecting the best mod...
My video on grid search: • How to find the best m...
My lesson notebook on StandardScaler: nbviewer.jupyter.org/github/j...
=== WANT TO GET BETTER AT MACHINE LEARNING? ===
1) WATCH my scikit-learn video series: • Machine learning in Py...
2) SUBSCRIBE for more videos: kzbin.info?su...
3) ENROLL in my Machine Learning course: www.dataschool.io/learn/
4) LET'S CONNECT!
- Newsletter: www.dataschool.io/subscribe/
- Twitter: / justmarkham
- Facebook: / datascienceschool
- LinkedIn: / justmarkham

[@dataschool]

*Are you new to Machine Learning?* Watch my video series, "Introduction to Machine Learning in Python with scikit-learn": kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@arunjohn492]

Sir what about dummy variable trap , When we use Column Transformer ?

[@dataschool]

Great question! See this video: kzbin.info/www/bejne/hIrXqKysrtt3e80

[@terryhenyo9216]

The Legendary Data Science guy is back!

[@dataschool]

Thank you for the warm welcome! 😄

[@GoredGored]

For beginners: When I tried to complete an ML project of say a simple model based on Logistic or Linear regression it used to take me about a month. As I was a beginner in Python, Pandas, SQL and the rest of it, I thought this will take me a long time to master and may be I am a late comer into this. But a year forward now and thanks to Data School, Sentdex, Krish naik, Statquest, Thinkful Webinar and more I am surprised that all I need is a day or less to complete these projects. Because of the meticulous analysis on Data School when I needed a deeper understanding that's where my gps leads me to. Thank you Data School.

[@dataschool]

You are so very welcome!

[@altunbikubra]

Your guideline does not only involves basic codes, but it actually involves very practical and useful functions. I want to sincerely thank you for your effort!

[@dataschool]

Thanks very much for your kind words!

[@rommeltito123]

Dayyyyuuummmm.......why did I not stumble upon ur videos earlier ????!!!!!!

[@dataschool]

😄

[@Anarchy977]

Fantastic tutorial! Great teacher, best Machine Learning teacher on youtube! Thank you!

[@dataschool]

Thanks so much!

[@sandeep1026]

I feel fortunate that I stumbled across this video. Very well articulated. Slows down pace, so that folks can hear, understand and digest. Most videos I come across, seem to rush through the contet before one can digest. Thanks for taking time and sharing your knowledge

[@dataschool]

Thanks very much for your kind words! 🙏

[@MrMehrd]

Thx kevin, one of best & simplest explanations of pipeline

[@dataschool]

Glad it was helpful!

[@liquid_absabs1334]

There is something about your explanations, that i just get it instantly. You deserve an award

[@dataschool]

You are too kind, thank you!

[@dataschool]

Yes, that is the role of the OneHotEncoder.

[@salonisamant5410]

Thank you for explaining the pipeline approach so well!

[@dataschool]

You're very welcome!

[@christianiheanacho4976]

You are a high quality TEACHER , thank you very much.

[@dataschool]

You are very welcome! 😄

[@sophiar5280]

Always love your step by step, clear lessons. Keep it coming.

[@dataschool]

Thank you!

[@tald747]

This is an excellent and simple explanation of this topic. I must say that you are a very talented in the way you teach! You choose your words in a way that emphasizes only the important and relevant staff. Thanks!!!

[@dataschool]

Wow, thank you!

[@quocanhhbui8271]

My god I love your detailed solution. Even my 5yo sibling can understand it. Wonderful. Definitely worth a subscribe.

[@dataschool]

Awesome! 🙌

[@horoshuhin]

thank you Kevin, very thorough explanation. I'm glad I found your channel. I like the way you teach.

[@dataschool]

Thank you so much! 🙏 That's great to hear!

[@fet1612]

00:58 1) It allows you to properly cross-validate a process rather than just a model. In other words, when you are doing cross-validation like cross_val_score, normally you just pass a model to it. Well, there are cases when that is not going to give you accurate results because you're doing the preprocessing outside of the cross-validation. So a pipeline, generally speaking, is useful because you can cross-validate a process that includes (a) *preprocessing* as well as (b) *model building*.

[@Freethinker33]

I was looking for clear explanation of Pipeline for a long time. You nailed it. Crystal clear explanation and understood by watching one time. Thank you.

[@dataschool]

You're so very welcome! 🙏

[@artyb3115]

Absolutely perfect and useful lessons! Thinking of becoming a patron member as I get a little more confident with ML

[@dataschool]

That would be awesome, thank you so much! You can join here: www.patreon.com/dataschool

[@420nyk]

Thanks, this helps a lot. Was scratching my head on pipeline and column transformer before this video. Also you got a very soothing voice and it helps to relax and really enjoy the learning.

[@dataschool]

Great to hear!

[@nishantchaudhary7528]

That was really something amazingly explained, I was looking for all these topics to understand. I got it in one go. Thanks a ton.

[@dataschool]

You're very welcome!

[@hieungotrung5411]

OMG!!! I’ve just started ML in kaggle for the past few weeks. Theres a lot of information to absorb but you teach us in the most understandable way and yet up-to-date question why we should use scikit instead of using dummies. This video is extremely helpful and informative. Thank you alot!!! Guess I gonna spend the rest of the day to watch all of your videos

[@dataschool]

Awesome! Glad to hear this was helpful to you 👍

[@harshitarawat8941]

Man I love you. I just love you. I love your videos. I love the way you explain things. I love the pace of you videos. I love everything. Thank you.

[@dataschool]

Thank you so much, Harshita! 🙏

[@PaulBillingtonFW]

Thanks, for this clear and well paced tutorial.

[@dataschool]

Glad it was helpful!

[@dhananjaykansal8097]

Nice to have u back sir. This session was so fruitful. Thanks a ton. Keep it up!

[@dataschool]

That's awesome to hear!

[@NoWhiteGullibility]

Perfect timing, was just searching on pipelines the other day. Would be great to follow-up by tacking on Gridsearch in this context.

[@dataschool]

That's awesome to hear! I will definitely cover grid search of a pipeline at some point - thanks for the suggestion!

[@jatinshetty]

yo! Mind blown with the amount of things i learnt from this. Please keep at it!

[@dataschool]

Thank you! You might like my scikit-learn tips: github.com/justmarkham/scikit-learn-tips

[@fahadkhankhattak8339]

thank you so much!!!!! it was very helpful. yours is the only channel i come running to for help whenever im stuck somewhere. rich conent!! keep sharing these wonderful thingss

[@dataschool]

Thank you so much!

[@sanaullahkhanhassanzai8432]

Thank you very much and welcome back after a long time. You are as good as gets when it comes to Machine Learning. You have made me learn a lot. I cant wait for videos on deep learning. I hope you ll come up with deep learning soon. Thanks again

[@dataschool]

Thanks very much for your kind words, and for your suggestion as well!

[@amitsharma8337]

THANK YOU for this tutorial! Was wandering around the web to solve unexpected errors that came by following, apparently, outdated tutorials. If I have landed up on this tutorial the very first time, it would have saved me around 4 hours of useless surfing. Thanks again

[@dataschool]

That's awesome to hear... glad I could be of help! By the way, I'll be launching a full course covering these topics (and more)... sign up here to get notified when it launches: scikit-learn.tips

[@chr1112]

you are the best tutor i have ever met , keep up the good work. Thank you

[@dataschool]

Wow, thanks!

[@Putinka1000]

Thank you for speaking slowly. It’s nice to listen to a non-English speaking person

[@dataschool]

You're very welcome! :)

[@amitblizer4567]

Very clearly explained and helpful video - Thank you!

[@dataschool]

Glad it was helpful!

[@ayyappahemanth7134]

Oh my god! after so much of exhaustive waiting another video came, which is far more useful than others for me! I just love your videos, the content was really useful in my real life, most of the youtube channels they just take the ideal ones which I might not encounter in my whole life! please do these videos regularly!

[@dataschool]

That is awesome to hear, thanks so much for your kind words! 🙏 Actually, I publish a new Q&A video every month for Data School Insiders at the $5 level: www.patreon.com/dataschool

[@jkore2554]

Thank you for this tutorial. I was working with logistic regression this week and was trying to figure out how to one hot encode for a categorical variable with hundreds of categories. I was getting 100% accuracy and precision so something wasn’t right. I’m going to try the steps that you outlined in this tutorial. Thanks.

[@dataschool]

Good luck!

[@honprarules]

Amazing explanation, as always!

[@dataschool]

Thank you!

[@salakkal]

Really great that you did a video like this . It just helped me a lot and I am really thankful for it brother . Keep going .

[@dataschool]

Thanks!

[@harshalkulkarni511]

Preprocessing with pipeline was complex topic to understand for me before watching this video. Thanks a lot for the video.

[@dataschool]

You're very welcome! Glad it helped 👍

[@aaqibsoomro5776]

You are a great teacher. Please make the tutorials or series for Data Visualization, In-Depth Data Analysis, and Cleaning, and Project Deployment, etc. Since after Learning Python and its libraries and ML, these are the next steps.

[@dataschool]

I have many more tutorials! Many of them are listed here: www.dataschool.io/launch-your-data-science-career-with-python/

[@abdelkaderkaouane1944]

Your explanation is very clear, thank you very much

[@dataschool]

You're welcome!

[@georgeognyanov]

God damn this video is good. I was struggling with column_transformer and pipelines till late last night. The options you suggest here are so much better and easier to understand for me. I am totally going through your "Introduction to Machine Learning in Python with scikit-learn" playlist soon. Thanks for putting this out!

[@dataschool]

You're very welcome! If you want to go deeper into this topic, you may want to check out my course: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@aimenbaig6201]

i just discovered your channel and i gotta tell you , you got a permanent subscriber here!!! LOVE YOUR TEACHING STYLE!!!!!!!!!!!!!!!

[@dataschool]

Thank you! 🙏

[@asimssheikh]

Impressive explanation, and logical approach to material presentation. You just got a new sub.

[@dataschool]

Welcome aboard!

[@sandeeppreetam]

Thank you good sir, this tutorial was better than many paid tutorials on Udemy. Blessed!

[@dataschool]

Glad it was helpful! 🙌

[@jobihara]

Thankyou dataschool, it was not only helpful, it was great, enlightening and awesome.

[@dataschool]

What a nice thing to say, thank you so much! 🙏

[@adarshr30]

After searching alot, i found this channel n i feel its best for me:)

[@dataschool]

Happy to hear that!

[@sowash2020]

You just gained another subscriber...this was super useful

[@dataschool]

Great to hear!

[@lovejazzbass]

Kevin, it's 5:20am Winston-Salem time and I am digging this. I was very confused. Thank you so much.

[@dataschool]

Excellent!

[@brandonbermudez9047]

Absolute goat bruh, really thankful for your content

[@dataschool]

Thank you!

[@trentjones6468]

Amazing video. You are an excellent instructor. Got yourself a new subscriber :)

[@dataschool]

Thank you so much!

[@krishkonnect814]

I just found solution to my problem after watching your video. Thanks a lot.

[@dataschool]

You're welcome!

[@David-fr7ee]

Great content, i am learning this in my college data science class. You did better than my professor!

[@CE-vd2px]

Are you undergrad or grad?

[@dataschool]

Thank you! 🙏

[@xinchenzou4558]

Thank you sir! You've really saved my life...

[@dataschool]

🙌

[@frankgiardina205]

Excellent! I was using the pandas dummies and your explanation of why pipeline and ohe is a better solution solves all the problems. thanks again

[@dataschool]

Glad it helped!

[@SaunakDey]

awesome explanation!! Thanks a lot

[@dataschool]

You're very welcome!

[@ramleo1461]

Hi, this will be very helpful.. Thank you for making this video!!

[@dataschool]

You are very welcome! 🙌

[@gyanendergandhar]

Thanks alot for this tutorial Kevin. It really saved me😅

[@dataschool]

Glad to hear that!

[@barulli87]

MIND BLOWN!!!! CV FOR A PROCESS!!! NOICE ONE!!

[@dataschool]

🤯

[@absar66]

Great ! Great ! Great! tutorial..many thanks Kevin

[@dataschool]

You're very welcome!

[@surfzion]

Extremely helpful, thank you so much !!!

[@dataschool]

Glad it helped!

[@kishanlal676]

Thank you for this amazing video. Please do some videos on feature selection and scaling techniques in python!

[@dataschool]

I'm hoping to cover feature scaling in a future video, but I do have a video about feature selection: kzbin.info/www/bejne/j5Kufph3oa2ap7M Hope that helps!

[@christianiheanacho4976]

I am enriched by this teaching.

[@dataschool]

Great to hear!

[@gisleberge4363]

Great example, educational.

[@dataschool]

Thank you!

[@1stophchr]

thank you very much, very clear video

[@dataschool]

You're very welcome! 😄

[@yeahzisue]

this is so helpful that I have to comment. great job. thanks a lot

[@dataschool]

Glad it was helpful!

[@Steven-se5jd]

just want to say thank you. I am a beginner and you teach much better than my professor.

[@dataschool]

Glad to hear I have been helpful! 🙏

[@eugenechew1476]

Why pay $900 at Uni when you can watch this amazing tutorial for free, and its wayyyy better!

[@dataschool]

Thanks! Stay tuned for a course that explores these topics is much more detail...

[@83vbond]

I paid $6000 :((

[@hichamamchtkou7343]

Thank you very much, it 's very interesting and by the way, it is exactly what i need in my current ML project.

[@dataschool]

That's great to hear! Good luck with your project 🙌

[@hichamamchtkou7343]

@@dataschool thanks 👍

[@brendensong8000]

I love it! Amazing tips!

[@dataschool]

Thank you!

[@victor-os9wq]

Thanks for such a detailed tutorial. I am working on a similar problem where I have multiple categorical features. In my dataset, the categorical variables has more than 90 possible values, as a result I am having an additional 121 columns when i use the Get.dummy, but I actually want just four levels. Please kindly advise me.

[@Takk6]

You are by far the best data science teacher on youtube. Can you make a video on creating your own custom transformers using it to modify your data, then using that custom transformer in a ColumnTransformer and a Pipeline?

[@dataschool]

Thanks for your suggestion! I'm working on a course that will likely cover that topic. Sign up here to get notified when it launches: scikit-learn.tips

[@oeb5542]

Just another amazing video. 😄

[@dataschool]

Thank you so much for your kind words! 😊

[@Pqj613]

It's a good tutorial for some reasons that you will explain later.:D

[@JainmiahSk]

Sir, just before 5 minutes I visited our channel to ask you the same question where it was difficult for me to encode multivariables in kaggles house prediction using advanced regression dataset. Fortunately and surprisingly you posted same. Thank you so much.

[@dataschool]

That's amazing! 🙌 I hope this video is helpful to you, and let me know if you have any questions!

[@JainmiahSk]

@@dataschool I have a problem with functions, I can't write custom functions in Python which is very important what to do sir?

[@dataschool]

@@JainmiahSk You can definitely write custom functions in Python!

[@TheAdrianPardo]

Thank you so much! You're the best! Please go over scaling when you have a chance :) Question: Is is ok to leave in all of the OneHotEncoded columns with this pipe approach? I believe you previously mentioned how it's best to drop one of the columns to prevent multicollinearity. Any way to do this within the pipe?

[@dataschool]

You are so kind, thank you! 😊 Yes, I plan to cover StandardScaler at some point. Yes, it is okay to leave in all of the one-hot encoded columns. However, the "drop" parameter for OneHotEncoder (new in scikit-learn 0.21) does allow you to drop one feature per category. Hope that helps!

[@ramleo1461]

Even I had the same doubt... Thank you for clarifying 😊

[@eatbreathedatascience9593]

This video is excellent.

[@dataschool]

Thank you!

[@MohammadrezaMokhtari-qh2yg]

amazing information. wow! thank you so much man.

[@dataschool]

You're very welcome!

[@AjayVerma-xi2us]

Very good, it cleared my many doubts

[@dataschool]

Great to hear!

[@abdoulayebalde2139]

A very nice video that save my life I can see it is well explained keep uploading

[@dataschool]

Thanks!

[@krishnaprasadbhat851]

mkayyyyy, awesome tutorial!!!

[@dataschool]

Thank you!!

[@anthonyhan6825]

Awesome job!

[@dataschool]

Thanks!

[@TheAstralftw]

Finally someone explained me properly what is columns transformer and why we use pipeline. I would like you to put your course to udemy , then i ll buy it 100% .. maybe on average you will sell each course for less price, but trust me, you are explaining this so good, you can sell tens of thousands of courses in few months , ... or in the case you have this on udemy , please provide me with the link!

[@dataschool]

Thanks for your kind words and your suggestion! I know that many students like Udemy courses, but my values as a course creator don't align with their business model, and so I'm not currently interested in publishing a course there. I prefer to offer courses directly to interested students. Thanks for understanding!

[@nguyenminhoan7882]

thanks you, waiting for more tutorials :3

[@dataschool]

You're very welcome! I will do my best to publish more!

[@pivotai525]

Simply the best!!

[@dataschool]

Thank you!

[@vincecarter7500]

thanks a lot for helping everyone out, was just wondering if you will be uploading more videos in the future

[@dataschool]

Yes! I just started posting again last week. Thanks for watching!

[@user-bt8ln1pp1k]

Very Well Explained..

[@dataschool]

Thank you!

[@zohrehvahdati787]

Thank you so much.😍😍🙏🙏👍👍 It helped me a lot.

[@dataschool]

Great to hear!

[@nowhere5111]

This video helps a lot👍👍👍

[@dataschool]

Great!

[@emilezg4496]

great content thank you sir

[@dataschool]

Thank you!

[@sihlengena5022]

Simply the best.

[@dataschool]

Thank you!

[@Narriz]

This is amazing.

[@dataschool]

Thank you! You might be interested in this course: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@12345shipreck]

You are 100x better than my ML course teacher at uni. GG bro.

[@dataschool]

Thank you! 😄

[@davidfullstone]

Your videos are amazing and are really helping with the last module on my MSc. I know there is no need to encode Pclass as it is an ordinal variable that is already ordered and you explained that really clearly. I notice also that you explain well about use cases and processing with regard to onehotencoder vs ordinalencoder in other videos. For marking/best academic marking practices in my module would you recommend creating a onehotencoder for my nominal and a ordinalencoder for my ordered data then piping both through make_pipeline? Thank you in advance :-)

[@prithasinha5378]

This is a great video! Thank you. Will you be showing how to do parameter tuning with pipeline?

[@dataschool]

Yes, I actually cover that in one of my courses: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@andrecouto2344]

Thank you very much man

[@dataschool]

You're welcome!

[@schuylerblasy2192]

This is a really interesting video. Column_transformer is sort of like a pipeline in itself. Kind of reminds me of vectotassembler in Spark/Pyspark.

[@dataschool]

Thanks Sky! One important difference is that ColumnTransformer stacks results side-by-side, whereas Pipeline feeds the output of one step to the input of the next step.

[@lourdesmartinez1486]

Hi! First I would like to thank you for this awesome video! Super well explained, super clear and veeeery useful. Thanks a lot! I have a question, does it makes sens to encode n times (cv=n in your cross val) the data set? I mean... using a pipeline is great for test purposes as you explained, but I am not sure that is necessary to use the entire pipeline (including encoding) for the cross val...but maybe I am missing something. Could you clarify this point? Thanks it advance for your comments !

[@dataschool]

Great question! Yes, it is critical that you cross-validate the entire pipeline (rather than just the model) so that the data preprocessing occurs within each fold of cross-validation. Doing the preprocessing prior to cross-validation can lead to data leakage, which means that your evaluation scores will be less reliable. This is a complex topic, but I hope that helps a bit!

[@joxa6119]

God this video answered my month unsolved question. God blessed you.

[@dataschool]

Great to hear!

[@salseid1033]

Your tutorial is informative as always. May you prepare a tutorial how to interprete model. Like 'Black Box' interpretation in RF. Thank you.

[@dataschool]

Thanks for your suggestion! I'll consider it for the future!

[@deweihu1003]

Nice video!

[@dataschool]

Thanks!

[@KVishya]

Hi Kevin, thank you so much for the wonderful explanation, could you also explain how to use GridSearch or RandomizedSearch along with Pipelines?

[@dataschool]

Great suggestion! I'm working on a tutorial that will be published on KZbin in late April. It will include that topic. Stay tuned!