Handling imbalanced dataset in machine learning | Deep Learning Tutorial 21 (Tensorflow2.0 & Python)

[@codebasics]

Check our Deep Learning Course (in PyTorch) with Latest, Industry Relevant Content: tinyurl.com/4p9vmmds Check out our premium machine learning course with 2 Industry projects: codebasics.io/courses/machine-learning-for-data-science-beginners-to-advanced

[@tugrulpinar16]

Those who are watching just recently, SMOTE function is "fit_resample" now. Also if you can't import imbalanced_learn properly, try restarting the kernel.

[@krishnaprabeesh2415]

Thank you

[@sonalganvir8334]

Will this work for categorical response too?

[@sanjaydubey8036]

@Ma Aleemit means n_jobs = -1, i.e. use all ur cores for processing

[@ABZein]

Thank you

[@iaconst4.0]

gracias amigo!!

[@magdalenawielobob9464]

Hi. You should perform under / over sample (including SMOTE) only on training data, and measure f1 on original data distribution (test data). Moreover, if you divide oversample data with train_test_split then you have no control over the distribution of duplicated items for test and train. Which means that you can have the same observation in both test and train, which means you test partially on the training set - that's why the results increase. So first divide into train / test, and then perform operations only on the training set, and the test set should be without any changes. Still, it's a very good tutorial, it's nice that you share your knowledge !!

[@charithaweerasooriya5941]

yes thats true

[@vineetkumarmishra2989]

yeah, we should never touch the test set.

[@MMSakho]

True.., it might will be overfit right?

[@Stenkyedits]

sad but true

[@nithinmanjunath3909]

@@MMSakho Yes you are right

[@tjbwhitehea1]

Hey codebasics, love this video series! I think there’s a pretty big mistake in the oversampling though. You upsample, then do train test split. This means that there will be overlapping samples in both train and testing data, so the model will have already have seen some of the data you are testing it on. I think you need to do your train test split then do the upsampling on the train data only.

[@shivi_was_never_here]

Yup, that's true. My professor said you should always oversample after splitting the data, and undersample before. If you oversample before splitting the data, your model will be in danger of overfitting. Yay, go me, commenting on a 3 year old comment!

[@danieltheone8081]

@@shivi_was_never_here this comment just helped me avoid this mistake. So, awesome, yay!

[@stanleypiche4705]

Thank you so much for sharing this interesting information about data transformation. I was training a neural network that gave an AUC of 0.85, after balancing the class with the SMOTE it reached 0.93 AUC. Obviously, the f1-score and accuracy also improved. Thanks!

[@MrAiblack]

The way you are introducing the information is very very excellent, thanks for sharing your knowledge and I'm happy to watch your video

[@tchintchie]

I always learn something new watching your videos. Thank you 🙏🏻

[@codebasics]

I'm so glad!

[@Rajdeepsharma1987]

Thanks for providing us the path and please keep doing the good work and don’t get upset by lesser views you are a true inspiration for all of us.

[@yogeshbharadwaj6200]

Only in this video looks like your patience was out of your control sir....huhaaaa....but still quality content delivery and great explanation....Tks a lot Sir....

[@manansharma4268]

Thank you very much for this video. This actually helps in solving real world scenarios.

[@codebasics]

:)

[@avisimkin1719]

nice video, pretty clear. I think there are 2 things that are missing though: 1) Doing the under/oversampling only on training data 2) You could have also choose a different operating point (instead of np.round(y_pred), taking a different threshold) , or just using AUC measure and not rounding at all, that could have been more indicative PS: SMOTE don't actually give any lift in AUC measure. you off just as well adjusted the threshold to y_pred>0.35 or something like that and get better F1 scores

[@maor940]

True. Good points!

[@MaximityL]

My thoughts exactly. Nice!

[@rohanpatnaik7348]

00:00 Overview 00:01 Handle imbalance using under sampling 02:05 Oversampling (blind copy) 02:35 Oversampling (SMOTE) 03:00 Ensemble 03:39 Focal loss 04:47 Python coding starts 07:56 Code - undersamping 14:31 Code - oversampling (blind copy) 19:47 Code - oversampling (SMOTE) 24:26 Code - Ensemble 35:48 Exercise

[@meriemkh4382]

the evil laugh at 22:28 😂😂

[@sandiproy330]

Tremendous respect sir, I love your tutorial. I sincerely follow your tutorial and practice all exercises that you provide. However, I went through some comments for this video lecture and found that people are suggesting to oversample/SMOTE the training sample only, and not to disturb the test sample (which I too believe is quite apparent, as this will avoid duplicate or redundant entry in training and test data set). Hence, separated out the train and test datasets first, then applied the oversample/SMOTE technique on the training dataset only. Unfortunately, the precision, recall, and f1-score are not increasing for the minority class. This is quite logical though. What I understood is, duplicate entry of the same sample in both the train and test dataset was the reason for that huge increase in minority class precision, recall, and f1-score in your case.

[@sandiproy330]

This happened when I tried the second exercise of the Bank customer churn prediction problem. Oversampling/SMOTE on train data gives around 0.51, 0.63, and 0.56 for precision, recall, and f1-score. When I follow your method for the Bank customer churn problem, the figures are 0.77, 0.90, and 0.83 respectively.

[@asiastoriesmedia519]

Thanks!

[@venkatesanr9455]

Thanks a lot, codebasics for all of your valuable and knowledgeable content

[@honeyBadger582]

i was actually doing the churn modeling project and this video popped up! thanks a lot :)

[@codebasics]

Glad I could help!

[@CarolynPlican]

Thank you. Very clear instruction and linked to Ann too, as I've only used with supervised ml.

[@shylashreedev2685]

Hats off to u Dhaval, Loved ur way of teaching and clearing my concepts, thank u so much

[@ybbetter9948]

Great presentation! I think I just needed SMOTE for my assignment but I liked how you explained every method.

[@paulowiz]

So fun the laugh at 22:31 hehe really cool video!

[@PriyankaDarshanam]

Note: The fit_sample() method has been replaced by the fit_resample() method in newer versions of imblearn

[@daxalakdawala427]

THANKS

[@gurkanyesilyurt4461]

Thank you again Dhaval. I really appreciate your efforts!!

[@sakalagamingyt3563]

31:40 the ANN function is using the same old X_test and y_test. I think that's why the accuracy is so bad.

[@aniljhurani8289]

Very interesting, amazing video...at 22:34 when using SMOTE method , smote.fit_sample(X,y) is now smote.fit_resample(X,y).

[@muhammadhollandi2586]

very helpful, your video makes everything easier ,thousand thumbs up for you 👍👍

[@codebasics]

Glad it helped!

[@GuilhermeOliveira-se1th]

You answered my question with only 4 minutes. Great! thank you!

[@codebasics]

Happy to help!

[@RoyalRealReview]

@@codebasics if we have ratio of data in 54% and 46%. Do we need balancing?

[@nurulfadillah1248]

Undersampling 7:34 Oversampling 15:04

[@flaviobrienza6081]

In my opinion the SMOTE part is not wrong, but it is tricky. Using SMOTE on the entire dataset will make the X_test performance much better for sure since it will predict values already seen. Instead, if you split your data before the SMOTE you can see that the performance improves, but not too much, it will not reach 0.8 if without SMOTE was 0.47. The X_test in the video could probably interpreted as the X_validation, and the testing data should be imported from other sources, or at the beginning the dataset should be divided into training and test, like on Kaggle.

[@siddharthkulkarni409]

I think we should first apply train test split and then over/under sample the train data.

[@tallandenglish]

Great stuff, but an error I believe. AT 31:07, in the ensemble method, you've used the function 'get_train_batch' to get X_train and y_train, but you're not redefining X_test and y_test

[@behrozjurayev5702]

🤩 love your tutorials brother

[@fahadreda3060]

Great video as usual sir , wish you more success

[@codebasics]

So nice of you. I hope you are doing good my friend fahad.

[@anuppudasaini6302]

Good experiments with different methods! How about Auto-encoders methods? You encode and decode all good data (customer staying per your example) within DNN, calculate its reconstruction error. Now you run customer leaving data in your model. If your error from customer leaving data is not within the reconstruction error (from your staying data), then you have detected an anomaly. What do you think?

[@harperjmusic]

Don't you want to apply SMOTE just to the training data, and leave the test data untouched?

[@lorizhuka6938]

True. Smote musst be appied after train test split.

[@MrMadmaggot]

@@lorizhuka6938 What about the others? Oversampling for instance.

[@sandiproy330]

Wonderful video. Great effort. Thank you.

[@codebasics]

Glad you enjoyed it!

[@raj-nq8ke]

Perfect explanation

[@codebasics]

Glad you think so!

[@johnmasalu8703]

Very useful and fruitful, big up

[@codebasics]

Glad it was helpful!

[@sksahungpindia]

Sir, Is there any better method from SMOTE for Class Imbalance? if yes please guide me...I am a Research Scholar (Doing Ph.D) from TOP 30 NIRF ranking institute. My area of research is classification problem in machine learning including dealing with imbalance data set. Thank you

[@JACKBLACK-jt8nw]

excellent approach very helpfull

[@aditya_01]

video is really helpful.Thanks for sharing.

[@codebasics]

Glad it was helpful!

[@raom2127]

Nice tutorial seen on this Topic Excellent Teaching....Could you please post Topics on supervised learning and unsupervised learning separately to know learn on sequense basis.

[@turalkerimov4022]

Best Teacher!!!!!

[@codebasics]

👍😃

[@josebordon46]

thanks for the great content, for the ensemble method could we use a random sample of the majority class (n=minority class length) then we could create more models for the vote

[@emmanouilmorfiadakis118]

JUST THE BEST

[@ashishdewangan485]

Hi @codebasics. I find your tutorial series very informative and interesting. I am learning a lot from your videos. I have a doubt in ensemble technique. While voting you are taking votes from three different predictions. But those predictions are not for the same data set. Is voting ensemble valid for such cases?

[@afeezlawal5167]

Same thought. Voting isn't ideal

[@peterjohngerero150]

Its a great tutorial! But i have a comment in the evaluation part. you applied Resampling first before splitting the data. So its possible that there's a leakage of data coming from the training to the test set. Right? thats why it has a equal prediction score. Its a good technique that you should split the data set first and then resample only the training set. Hope this helps. Thanks

[@vanajagokul5937]

Thank you so much. It was very informative.

[@codebasics]

Glad it was helpful!

[@spicytuna08]

awesome. cannot thank you enough

[@AlgoTradeArchitect]

Thank you for your sharing.

[@muhammadbasilkhan1829]

thanks for these good vide os these are very help full for me

[@vinodkinoni4863]

u r awesome teacher plz stay with us long live

[@codebasics]

thanks for your kind wishes Vinod

[@ubannadan-ekeh7781]

This is very insightful... thank you. Please can you do a video on Click through rate prediction

[@codebasics]

sure

[@sararamadan1907]

Great explanation

[@farhodkalonov9370]

Thank you so much and appreciate for your work.

[@siddharthsingh2369]

Could someone elaborate a little bit on how exactly data is getting overlapped. I see many people saying to first split data and then sample it, will it work because here in this video we are dividing class 0 and 1 well in advance and then combining the data. I am going through many comments on this issue and having a hard time to figure this out.

[@MrMadmaggot]

Did u manage to figured it out

[@DJ-jf4qg]

In over sampling minority class By Duplication if we duplicate minority class then both classes will have equal samples After that we use train-test -split which randomly selects samples. The problem is those duplicate samples will be present in training samples as well as testing samples thus increasing Precision,F1score and all of those. Here is the overlappping

[@dipankarrahuldey6249]

I think there's also a risk of overfitting the model when using SMOTE, as the synthetic data points might look like test data points(unseen).

[@MMSakho]

That's true. Especially if the data is in text

[@MrMadmaggot]

@@MMSakho Anyone managed to know if that's truth?

[@mprasad3661]

Great explanation bro

[@codebasics]

Glad you liked it

[@towhidsarwar1915]

sir, I am following your deep learning playlist. please make a video on cross validation with keras for neural network.

[@codebasics]

sure

[@emanal-harbi2004]

thanks, amazing illustration , do these methods work with multi-class labels ( means the lable column may contain over 10 labels)

[@omeryalcn5797]

Thanks for sharing, but i think, there is a problem for test metric. Because you use processed data for training( oversampling etc., that is okay ) but you can not use same preprocessed data for testing, because in real state you can not know test data target, so you can not use imbalanced technics. Firstly you should seperate data and only apply implanced process for train data and test without preprocessed test data.

[@soumyadev100]

Seems, we should not calculate accuracy on train sample, for oversampling it is pretty obvious that precision recall will improve. We need to test the accuracy on test sample, where we artifically have not increase or decrese the number of samples.

[@sergiochavezlazo5362]

Hi! Why dont directly use the train_test_split with the stratify argument? Thank u!

[@riazrahman7147]

Thank you so much.

[@mitalikatoch9404]

Hey, great video. Can you also make one video on how to handle the class overlapping (that too in imbalanced binary classification)?? Thank you

[@rohitkulkarni9038]

Which is the Best method to do the sampling before Spiting the dataset or After Splitting the dataset

[@prachi6160]

Can we use variational auto encoder for synthetic data generation in case of minority class?

[@dakshbhatnagar]

Hey Dhaval. Great Video however I have a question. Will using class_weight parameter in Tensorflow and assigning the values based on the occurrence of the classes create any sort of bias towards some classes?? Can class_weight be helpful for handling the imbalance and not doing any sampling of any kind??

[@abhaygodbole9194]

Hello Dhaval, Very Nice explanation.. Does SMOTE work for highly imbalanced data like I have data set where one class has less than 1% representation in the distributions ? Please clarify

[@ДаниилК-ь1ц]

Great video ! i'll thank you with subscription

[@fattahmuhammadtahabi945]

Really helpful. Could you please tell whether oversampling strategy is okay if we do cross-validation instead of train-test-split?

[@sksahungpindia]

Sir, please clear my doubt. in method-2 ie Oversampling when we use train_test_split method the precision,recall and f1-score value is not look realistic because my test data is not unique (means trained data is already is in test data because of oversampling). please clarify? Thanks

[@piyushdandagawhal8843]

True, when you over sample there is a good chance that there will be data leakage. It would be helpful if you split the data and then oversample the train data to avoid any influence on the result.

[@sksahungpindia]

@@piyushdandagawhal8843 Thank you Piyush. Please suggest me some research direction on Handling imbalanced data set in machine learning and Deep Learning. I am a full time research scholar so your suggestions mean a lot for me. Thank you

[@RGB_321]

I am getting error Failed to convert a NumPy array to a Tensor (Unsupported object type int).

[@AmarPalSingh-tn3sh]

does this approach work for more than 2 categories in Target variable?

[@Nick-tt9lh]

do we need to check for imbalance for unsupervised learning problem or clustering problem?? if yes, why and how??

[@sabinazaman3512]

I was trying to implement smote.fit_resample(X,y). But got this error "'NoneType' object has no attribute 'split'". Couldn't find solution. Can anyone help?

[@tirthpatel3491]

Thanks for sharing it. I am wondering that how we can treat imbalance dataset of time series ? Can all mentioned techniques in video be performed on timeseries data?

[@naveenkumarmangal9653]

In general, it depends on type of data. Most of the imbalanced time-series dataset can be handled using SMOTE approach or combination of SMOTE with ENN/TOMEK.

[@ariouathanane]

Please some one can explain me, why in this example (on video) the accuracy and loss frequently changed? is this an overfitting?

[@otsogileonalepelo9610]

I also had a similar observation in all videos in this series

[@harshalbhoir8986]

Thank you sir

[@1bitmultiverse]

Legendary Quotes : 17:40 😂👍

[@bheemeshg4823]

After balancing the dataset may I know what values can be placed in that place

[@rasputinpootin]

You talked about Focal Loss but didn't show the practical application of it. Is there another video on Focal Loss?

[@anshi6205]

Thankyou so much🌈🌈

[@codebasics]

You’re welcome 😊

[@yogeshwarshendye4857]

Sir, can I use the methods used in this tutorial for training my image classification model or should I use augmentation for that purpose?

[@fonyuyborislami8034]

I think for image classification, augmentation is a better approach.

[@mayankseth1235]

Do you have video for imbalanced data for text classification problem. Please suggest.

[@tirthadatta7368]

can anyone give a solution of SMOTE memory allocation error problem. maybe many of u say that use premium GPU but it's too costly. Is there any other solution for solving this problem??

[@aomo5293]

Is it the same process in multi label classification ?

[@poharry2634]

Can we apply same technique if we have more than 2 classes?

[@amins6695]

Amazing video. One question. What if I use under/over sampling and accuracy or precision decrease? Single or combined under/over sampling methods let us to use features for further methods, for example, training multiple weak learners and then use ensemble methods. Is it possible for ensemble resampling methods?

[@halafazel2745]

awesome

[@ravikanthr34]

Sir I had a python coding implementing deep neural.netowrk on Kdd dataset can.you explain the coding toe in.a.gmeet session forever I will be.indebted to you thmq

[@mukulkatkar6170]

22:27 Crazy 🤣

[@roshanpeter9904]

In the ensemble method code, is it okay to split the data into batches first and then apply the train_split and train it for each, and then take the majority?

[@subhamsekharpradhan297]

Sir when I ran the code I got this error: AttributeError: module 'matplotlib' has no attribute 'get_data_path' what can be done for it?

[@faezehfazel224]

If we have imbalanced dataset but still get good F1-score, should we still be concerned about the data being imbalanced and use one of those techniques or not?

[@TanviambadasBamrotwar]

Thank you

[@NguyenNhan-yg4cb]

you look so sleepy bro, just make sure you stay alerty to deal with any troubles, just kidding man lol. Best wishes for your contry

[@annperera6352]

Hello Sir .i was looking everywhere for class imbalance problem.Thanks a lot for this video. Do you have any videos for implementing rule based classification?

[@abhishekprakash9803]

one more question suppose we build a model fraud detection based on datasets....like 40% defaulter and 60% non defaulter.......what happend if we passed different datsets ,different distribution...diffrent size,quality......new datsets approx...70% deafulter 30 % non defaulter.........so how we can overcome this problem.........we build two models ,we combine two datsets....to build one model.....plz commnet ...

[@karangadgil9847]

can we use SMOTE while working with audio dataset ?