Machine Learning Tutorial Python - 6: Dummy Variables & One Hot Encoding

[@codebasics]

Folks, here's a link to our bootcamp for learning AI and Data Science in the most practical way: tinyurl.com/395u4mnm

[@celestineokpataku]

I have watched only 4 mins so far i had to pulse and write this comment. I will say this is one of the best tutorial i have seen in data science. Sir you need to take this to another level. What a great teacher you are

[@codebasics]

That for the feedback my friend 😊👍

[@chitz7435]

100% aligned...am doing an external course but have to refer to ur session to understand the topic in external course...amazing effort..

[@venkatesanrf]

Hi, Your explanation is very simple and effective Ans for practice session A)Price of Mercedes Benz -4Yr old--mileage 45000= 36991.31721061 B)Price of BMW_X5 -7Yr old--mileage 86000=11080.74313219 C) Accuracy=0.9417050937281082(94 percent)

[@ANIMESH_JAIN04]

Same bro

[@fathoniam8997]

same bro.... thx for replying so that i can check my results

[@msaad_313]

ValueError: X has 3 features, but LinearRegression is expecting 4 features as input. I am getting this error. plz can you provide your code.

[@sreenufriendz]

Anyone can be a teacher , but real teacher eliminates the fear from students .. you did the same !! Excellent knowledge and skills

[@codebasics]

Sreenivasulu, your comment means a lot to me, thanks 😊

[@jhagaurav8292]

Sir pls continue your machine learning tutorials ,yours tutorials are one of the best I have seen so far .

[@codebasics]

sure Gaurav, I just started deep learning series. check it out

[@samrahafeez5001]

@@codebasics Kindly explain the concept of dummies in deep learning as well

[@TheSignatureGuy]

For anyone stuck with the categorical features error. from sklearn.compose import ColumnTransformer ct = ColumnTransformer([("town", OneHotEncoder(), [0])], remainder = 'passthrough') X = ct.fit_transform(X) X Then you should be able to continue the tutorial without further issue.

[@muhammadhattahakimkeren]

thanks bro

[@fatimahazzahra6181]

thanks a lot! it helps

[@souvikdas3189]

Thank you brother.

[@Ran_dommmm]

Hey, thank for the code. I tried using your code but it gives me an error, despite of converting it (X) to an array, it gives me this error. " TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array. "

[@TheSignatureGuy]

​@@Ran_dommmm I know you said "despite converting X to an array", but just double check you have used the .toarray() method correctly. The error message seems pretty clear on this one. This function may help confirm that a dense numpy array is being passed. import numpy as np import scipy.sparse def is_dense(matrix): return isinstance(matrix, np.ndarray) Pass in X for matrix and it should return True. Good luck fixing this.

[@noubaddi8567]

This guy is AMAZING! I have spent 2 days trying decenes of other methods and this is the only one that worked for my data and didnøt come as an error, this guy totally saved my mental sanity, I was growing desperate as in DESPERATE! Thank you, thank you, thank you!

[@codebasics]

I am glad it was helpful to you 🙂👍

[@vaishalibisht518]

Wonderful Video. This so far the easiest explanation I have seen for one hot encoding. I have been struggling from very long to find a proper video on this topic and my quest ended today. Thanks a lot, sir.

[@Genz111-o4r]

I was confuse from where to start studying ml and then my friend suggested this series.... It's great :-)

[@rishabhjain7572]

any other courses or source you are following? and any development you have begun ?

[@sauravmaurya6097]

want to know how much this playlist is helpful? kindly reply.

[@carti8778]

@@sauravmaurya6097 its quite helpful if u are a beginner. Beginner in sense of {not from engineering or programming background }. U can accompany this with coursera’s andrew ng course.

[@carti8778]

@@sauravmaurya6097 if u already know calculus and python programming (intermediate level) , ML would feel easy . After doing this go to the deep learning series bcz thats what used in industries.

[@tech-n-data]

Your ability to simplify things is amazing, thank you so much. You are a natural teacher.

[@ymoniem1]

you really made it very easy to understand such new concepts, Thanks a lot starting from mint 12:30 about OneHotEncoder . Some udpates in Sklearn prevent using categorical_features=[0] here is the code update as of April 2020 from sklearn.preprocessing import OneHotEncoder from sklearn.compose import ColumnTransformer columnTransformer = ColumnTransformer([('encoder', OneHotEncoder(), [0])], remainder='passthrough') X = np.array(columnTransformer.fit_transform(x), dtype = np.str) X= X[:,1:] model.fit(X,y) model.predict([[1,0,2800]]) model.predict([[0,1,3400]])

[@petermungai5508]

The code is working but give a different prediction compared to dummies

[@petermungai5508]

Plus my X is showing 5 column instead of 4

[@petermungai5508]

I was entering the 0 and 1 wrongly. I am getting the same answer thank you for the code

[@rameshkrishna1956]

thanks buddy

[@shrutijain1628]

this ML tutorial is by far the best one i have seen it is so easy to learn and understand and your exersise also helps me to apply what i have learn so far thank you.

[@codebasics]

Glad it helped!

[@tushargahtori1570]

Even in 23 your video is such a relief..kudos to your teaching.

[@bandhammanikanta1664]

First of all, 1000*Thanks for sharing such content on youtube.. I got an accuracy of 94.17% on training data.

[@codebasics]

Bandham, I am glad you liked it buddy 👍

[@mk9834]

I was shocked after the first 5 minutes of the video and have never thought it would be so easy and fast! Thanks ALOT1

[@codebasics]

Miyuki... I am glad you liked it

[@programmingwithraahim]

15:50 write your code like this: ct = ColumnTransformer( [('one_hot_encoder', OneHotEncoder(categories='auto'), [0])], remainder='passthrough' ) X = ct.fit_transform(X) X Ok so it will work fine otherwise it will give an error.

[@AxelWolf26]

what is the use of this " (categories='auto') " and " 'one_hot_encoder' "

[@jollycolours]

Thank you, you're a lifesaver! I was trying multiple ways since categorical_features has now been depreciated.

[@adilmajeed8439]

@@jollycolours correct, the categorical_features parameter is deprecated and for the same following are the steps needs to be followed; from sklearn.compose import ColumnTransformer ct = ColumnTransformer([('one_hot_encoder', OneHotEncoder(), [0])], remainder='passthrough') X = np.array(ct.fit_transform(X), dtype=float)

[@snom3ad]

This was really well done! Kudos to you! It's hard to find clear and concise free tutorials nowadays. Subscribed and hope to see more awesome stuff!

[@ZehraKhuwaja65]

I must say this is the best course I've come across so far.

[@hiver6411]

the god of data science......Amazing explanation sir..kudos to your patience in explanation

[@codebasics]

Glad it was helpful!

[@shadabtechno]

your are the best teacher on youtube , i have never seen before

[@abhinavb717]

I am getting 84% accuracy without encoding variable, but after encoding i am getting 94% accuracy on model. Thank you for your teaching. Doing great Job

[@tanmaykapure81]

This is the best machine learning playlist i have came across on youtube😃👍, Hats off to you sir.

[@HashimAli-tz8fw]

I achieved the same result using a different method that doesn't require dropping columns or concatenating dataframes. This alternative approach can lead to cleaner and more efficient code df=pd.get_dummies(df, columns=['CarModel'],drop_first=True)

[@codebasics]

Exercise solution: github.com/codebasics/py/blob/master/ML/5_one_hot_encoding/Exercise/exercise_one_hot_encoding.ipynb Everyone, the error with catergorical_features is fixed. Check the new notebook on my github (link in video description). Thanks Kush Verma for giving me pull request for the fix.

[@urveshdave1861]

Thank you for the wonderful explanation sir. However I am getting an error as __init__() got an unexpected keyword argument 'catergorical_features' for the line for my code onehotencoder = OneHotEncoder(catergorical_features = [0]). Is it because of change of versions? what is the solution to this?

[@bishwarupdey10]

_init__() got an unexpected keyword argument 'categorical_features' sir I get this error when I specify categorical features

[@sejalmittal1326]

@@urveshdave1861 Have you got any answer for this? I am having the same error

[@sejalmittal1326]

@@urveshdave1861 okay .. i will do that. thanks

[@tanvisingh9298]

@@urveshdave1861 Hey I am also getting the same error. how did you resolve it?

[@omharne1386]

I will say this is one of the best tutorial i have seen in ML

[@phil97n]

I'm reading a textbook that has an exercise to study this same dataset to predict survived. I just finished the exercise from the book - I can't seem to go past 81% score. Thanks for your awesome explanation

[@wangangcwayi9420]

You have gift of explaining things even to the layman. Big Up to you

[@codebasics]

Thanks a ton Wangs for your kind words of appreciation.

[@ankitparashar7]

Merc: 36991.317 BMW: 11080.743 Score: 94.17%

[@codebasics]

Your answer is perfect Ankit. Good job, here is my answer sheet for comparison: github.com/codebasics/py/blob/master/ML/5_one_hot_encoding/Exercise/exercise_one_hot_encoding.ipynb

[@vishalrai2859]

thanks for posting the answer bro

[@mutiulmuhaimin9156]

Could we upvote this comment to the top? Been looking for this for quite some time now. This is important, and this comment matters.

[@Augustus1003]

@@codebasics I used pandas dummy variable instead of using onehotencoding, because it is too confusing.

[@clashcosmos4641]

Got the same answer using OneHotEncoder after correcting tons of errors and watching videos over and over.

[@himanshusingh-vt9do]

my model score 94% Accuracy .Thankyou sir for amazing video.

[@ZOSELY]

I wish I could give this videos 2 thumbs up! Great explanation of all the steps in one-hot encoding! Thank you!!

[@datasciencewithshreyas1806]

One of the best explanation for Encoding 👌👍

[@codebasics]

Glad it was helpful!

[@maruthiprasad8184]

For Mercedec benz I got 51981.26, for BMW i got 39728.19 & score is 94.17% . Thank you very much to make ML easy.

[@weshallneversurrender]

The Data Science GOAT! One day I will send you a nice donation for all that you have contributed to my journey sir!

[@hamzazidan6093]

Iam here from 2024 after 6 years and I want to say that this playlist is wonderful! I hope that you update it because there're many changes in the syntax of sklearn now

[@codebasics]

Hey next week I am launching an ML course on codebasics.io which will address this issue. It has the latest API, in depth math and end to end projects.

[@AruLcomments]

You are doing a wonderful job, people like you inspire me to learn and share the knowledge i gain. It is very useful for me. All the best.

[@geekyprogrammer4831]

This is really the best series to get started with ML

[@shinosukenohara.123]

How are u starting?

[@codebasics]

Glad it was helpful!

[@geekyprogrammer4831]

@@shinosukenohara.123 I am watching this channel, Krish Naik and Andrew NG course on Coursera

[@gokkulkumarvd9125]

How can I like this video more than 100 times!

[@codebasics]

I am happy this was helpful to you.

[@vishwa4908]

Awesome, you're explaining concepts in very simple manner.

[@codebasics]

Vishwa I am happy to help 👍

[@bharathdwarakanath1587]

The label encoding done for the independent variable column, 'town' in the second half of the video, I think, isn't needed. Instead just doing One Hot Encoding is enough. Wonderful contribution anyway. Thanks!!

[@loycewaihiga6707]

I agree

[@shekharbabar2496]

the best video series on ML sir ....Thank you very much sir....

[@timse699]

You teach with passion! thank you for the series!

[@NoureddineBahi]

Think you very much...wonderful work..special think from Morocco in north of Africa

[@nationhlohlomi9333]

A PLACE TO RUN TO WHEN ONE IS STUCK, THANK UOU SO MUCH SIR

[@srinivasreddy1709]

Hi Dhaval, your explanation on all the topics is crystal clear. Can you please make videos on NLP also

[@piyushjha8888]

model.predict([[45000,4,0,0]])=array([[36991.31721061]]), model.predict([[86000,7,0,1]])=array([[11080.74313219]]), model.score(X,Y)=0.9417050937281082. Thanks sir for these exercise

[@farjadmir8842]

I also got them correct. Sir, this course is amazing. You have made it so easy to understand.

[@codebasics]

Glad to hear that

[@deekshithkumar3234]

superb and precisely explained

[@codebasics]

Thank you 🙂

[@leooel4650]

Mercedes = array([[36991.31721061]]) BMW = array([[11450.86522658]]) Accuracy = 0.9417050937281082 Thanks for your time and knowledge once again!

[@mallikasrivastava]

Your videos are awesome

[@codebasics]

Glad you like them!

[@rooshanghous6912]

This is an amazing tutorial! saved me so much time and brought so much clarity!!! Thank you!

[@istihademon1427]

Highly Qualitative.

[@elinem5311]

thank you, this helped me so much with multivariate regression with many categorical features!

[@jayshreedonga2833]

thanks sir nice lecture sir you are really a great teacher you teach everything so nicely even tough thing becomes easy when you teach thanks a lot

[@debaratighatak2211]

I learned a lot from the exercise that you gave at the end of the video, thank you so much sir!

[@prasadjoshi8213]

Hi sir !! Most easier way u teach ML. Thanks a lot!!!. I m going through ur videos and assignments. I got the answer for merce: 36991.31, BMW:11080.74 & model score :0.9417. The Model score is 94.17%. My QUE is how to improve the Model score ??? Is there any way to apply the features?

[@ramanandr7562]

Thank you sir🎉. You made my ML Journey Better.. 🤩

[@late_nights]

If anyone got struck at One hot encoder at 16:26 then type this command and execute pip install -U scikit-learn==0.20

[@dhananjaypatel3538]

Thanks 😃

[@kketanbhaalerao]

stuck and still not executed using your solution

[@manasaraju8552]

difficult topics are easily understood, Thank you so much for the content sir

[@MrArunlama]

I was learning through a paid course, and then I had to come here to understand this concept of dummy variable.

[@asamadawais]

Simply excellent explanation with very simple examples!

[@scriptfox614]

The import linear regression statement lol. Amazing tutorial. :D

[@Adnan25048]

That's a great tutorial of one-hot encoding. I was unable to find a complete example anywhere. Thanks for sharing.

[@codebasics]

Thanks Adnan for your valuable feedback

[@preetipisupati2308]

Thanks for the excellent video.. but due to the recent enhancements, ColumnTransformer from sklearn.compose is to be used for OneHotEncoding.

[@codebasics]

Preeti, can you give me a pull request.

[@sarafatima2252]

definitely one of the best videos to learn from!

[@regithabaiju]

Your tutorial video is helping so much for knowing more about ML.

[@codebasics]

I am happy this was helpful to you.

[@mapa5000]

You make it easy with your explanation !! Thank you !!

[@komalsunandenishrivastava9211]

That image on one hot encoding 🤣🔥

[@ayushmanjena5362]

15:50 write this code from sklearn.preprocessing import OneHotEncoder from sklearn.compose import ColumnTransformer ct = ColumnTransformer([('town', OneHotEncoder(), [0])], remainder = 'passthrough') x = ct.fit_transform(x) x

[@felixgallo5132]

They're basically the same however pd.dummy variables are easier to use. Thank u, sir.

[@codebasics]

yes I agree

[@Dim-zt5ei]

Great videos! Unfortunately it becomes harder and harder to code in the same time as the video because there are more and more changes in the libraries you use. For example sklearn library removed categorical_features parameter for onehotencoder class. It was also the case for other videos from the playlist. Would be great to have the same playlist in 2022 :)

[@codebasics]

Point noted. I will redo this playlist when I get some free time from tons of priorities that are in my plate at the moment

[@Dim-zt5ei]

@@codebasics Thank you for the reply and again : Great job for all the quality tutorials!

[@cahitskttaramal3152]

Thank you for wery well explained tutorial. I have one question though, you are training all of your data here and yet model score is only 0.95. Why is that? It must be 1. If you were to split your data and train it would make sense but your case doesn't. What am I missing here?

[@codebasics]

Alper, It is not true that if you use all your training data the score is always one. Ultimately for regression problem like this you are trying to make a guess of a best fit line using gradient descent. This is still an *approximation* technique hence it will never be perfect. I am not saying you can never get a score of 1 but score less then 1 is normal and accepted.

[@thanusan]

Excellent video - thank you!

[@SrinivasA-vk7if]

Excellent video.., thank you so much.

[@purnanandabaisnab2856]

nice teaching, really outstanding thanks a lot

[@flamboyantperson5936]

Please make regression video using preprocessing library with standaridization and normalization variables

[@armagaan007]

Wait wait... I don't see the point 😕 The first half of the video does the same thing as one hot encoding(the second half of video)but second half is more tedious and takes more steps Then why not use the pd.get_dummies instead of onehotencoding??? What's the advantage of using onehot?

[@codebasics]

I personally like pd.get_dummies as it is convenient to use. I wanted to just show two different ways of doing same thing and there are some subtle differences between the two. Check this: stackoverflow.com/questions/36631163/pandas-get-dummies-vs-sklearns-onehotencoder-what-is-more-efficient

[@armagaan007]

@@codebasics thank you :]... btw you make grt videos

[@mohammadismailhashime5239]

Very nice explanation, appreciated

[@infinity2creation551]

Dil jeet liya , yahi khoj rha tha

[@pranavakailash8751]

This helped me a lot in my assignment, thank you so much code basics

[@codebasics]

Glad it helped!

[@rachitbhatt40000]

This module makes my code hot!

[@leelavathigarigipati3887]

Thank you so much for the detailed step by step explanation.

[@codebasics]

Glad it was helpful!

[@chamangupta4624]

Beautiful explanation, very helpful

[@nelizaat]

If anyone is interested, we can also skip the label encoder when using column transformer altogether by using the below : x=df[['town','area']].values y=df['price'].values from sklearn.compose import make_column_transformer ct = make_column_transformer( (OneHotEncoder(categories='auto'), [0]), remainder="passthrough" ) X=ct.fit_transform(x) X = X[:, 1:] model.fit(X, y)

[@codebasics]

Thanks neenu for the tip. The notebook in video description is actually updated to make use of column transformer.

[@nelizaat]

@@codebasics I am sorry I did not check that. Thank u sir for your videos, words are not enough to convey my gratitude for sharing your expertise to all.

[@shylashreedev2685]

Superb and very simple way of explaining, really helped me a lot , sincerely i will solve all ur exercises, can u pl help me, as i m not able to download the Dataset for the exercises given, i am preparing the .csv file and solving it.

[@jollycolours]

Click on the link of the CSV file (on github), click on the "RAW" button on the top right and you'll be redirecte to a new window, right click anywhere on the new window and "save file" + name your file with .csv at the end. Else you can always recreate the dataframe using pd.DataFrame All the best.

[@indrakumari1854]

Sir, very nice explained

[@codebasics]

Glad it was helpful!

[@Jobic-10]

❤🎉🎉 Thank you. You earned a subscriber

[@claude-olivierbatungwanayo9059]

Excellent as usual!

[@dineshgaddi1843]

First of all thank you for making life easier for people (who want to learn Machine Learning). You explain really well. Big Fan. When I was trying to execute categorical_features=[0], it gave an error. It seems this feature has been depreciated in the latest version of scikit learn. Instead they are recommending to use ColumnTransformer. I was able to get the same accuracy 0.9417050937281082. Another thing i wanted to know, when you had initially used label encoder and converted categorical values to numbers, why we specified the first column as categorical, when it was already integer value ?

[@isaackobbyanni4583]

Thank you for this series. Such great help

[@codebasics]

Glad it was helpful!

[@annette4718]

This was ridiculously helpful. Thank you so much!!

[@codebasics]

Netté, I am glad you liked it

[@uvinodh90]

Thanks for the excellent tutorial.... I see there is a decrease in score between this and the exercise data. Maybe due to an extra column in exercise data ? With increase in columns on X, Will the linearRegression score decrease ?

[@AbdulSamiasm]

thanks for updating Eexerce code for oneHotEncoding

[@swaruppanda2842]

nicely explained👌

[@brijesh0808]

@13:20 we need to do : dfle = df.copy() ? because otherwise changes in dfle will reflect back to df Thanks :)

[@adarshdubey1784]

Yes u r right

[@jayasreecarey7843]

Many Thanks ! Great Explanation :)

[@giovannaluciagc]

Thank you! it was really well explained

[@richard_shaju]

You are a Gem

[@satheeshkumar6849]

Thanks Bro for your help.I see there is difference in prediction using below method. 1)I followed your method and i got score of 94.17 2)I used X_Train,X_Test,y_Train,y_Test=train_test_split(X,y,test_size = 0.2,random_state=12) lm = LinearRegression() lm.fit(X_Train,y_Train) i got score of 91.7.Can you please suggest when we need to use which one?

[@amalsunil4722]

Bro, that's cuz in the first part you trained and tested using your training data, therefore, the MAE (mean absolute error) called bias is less. On the other hand, when you test your data with a new set of values(test data) your MAE called variance will be higher.

[@6223086]

thank you so much it has helped me in my work

[@codebasics]

Hey Eugene, I am glad to hear that it helped you in your work. Stay in touch for more videos and share our channel if you really find it worth.

[@Aaron_duckroast]

Please keep making ♥️ we love u