Machine Learning Pipelines A-Z | Day 29 | 100 Days of Machine Learning

[@geekyprogrammer4831]

I think this channel is better than Krish Naik's channel!

[@siddhartharaja9413]

both are amazing,please don't make any comparisons,

[@GhostRider....]

yes bro,even i think this is more in depth and very clear

[@smile93]

Indeed.....I have purchased tech neuron and definitely verify that

[@ankurkumar1466]

I also purchased tech neuron but it’s of no use…he’s better then kris naik…

[@whitepanda95]

Yes, Nitish isn't in hurry unlike krish😅

[@ParthivShah]

Thank You very much Sir, You could have taught the same thing in 10 minutes but you took efforts to saw us consicounses of not using pipes so that we can understand the importance of the same. Hats off to your work. Thank youuuuuuuuuuuuuuuuuuuuuu.

[@sam-mv6vj]

This video is literally gold brother for fresher Data scientists

[@YashPrajapati-t9x]

Making ML much much easier for every student out there....really you are an awesome teacher explained every procedure and techniques so clearly.... thankyou so much sir❤❤ best videos

[@LuckyRooopa8]

I don't know how are u, but thanks brother I loved your videos.... you make not only day every day to me

[@vaibhavchaudhary5571]

Nitish Sir, No one is better than you in teaching ML topics.

[@ashishmishra7506]

Thank a lot for your efforts, one of the best video I was looking for from long time.

[@ankitbhatia5998]

Best Channel to learn data science.... :)

[@Aman-tr5il]

You are a legend buddy. Brilliant & thanks a billion.🙏🙏🙏🙏🙏

[@yudhveersingh655]

Nice Explaination

[@tusharbedse9523]

Ekdum sahi bhai.. shiddat se video banaya aapne 😁😁😁

[@narendraparmar1631]

Thanks

[@aadisharma3514]

Thanks for making this video its really help me....

[@syedaafreen1075]

Amazing

[@rashidsiddiqui4502]

thank u sir

[@arpitchampuriya9535]

20:48 with pipeline

[@Code-Pedia]

No change in accuracy after removing the feature selection step

[@harithapa9262]

Pipeline ko agar khud se explore karna hai toh ek baar model ko streamlit ya phir flask me dalo phir tumhe pata chalega ki bina pipeline ko use kiye kitna code likhna padta hai aur agar pipeline use karte ho toh easily ho jata hai😊

[@Findout1882]

great job

[@Achu12321]

Thank you a lot ❤

[@gamingwithdivyansh7788]

sir ham train_test_split karne se phle sare transform kyu nhi krte?

[@hitendrasingh01]

great

[@arri5812]

😍😍😍😍😍😍😍

[@harshkondkar3193]

I gained more knowledge from this video than I have from an entire semester worth of classes in my university. Keep up the good work Sir!!

[@readbhagwatgeeta3810]

Sir....tear in my eyes 🥺🥺 I thought problem in my side that I am dumb...But what you taught all are like diamond 💎 No one can take so much time even 50000 ₹ course to make concept clear.... But now I am sure the problem was not from my side or more pricisely, best teacher do not need bright student to teach they can make the dumb student bright😄🤗

[@tathagatasharma]

There are no bad Students , only bad teachers. ( quote from Karate Kid) you were involved with wrong teachers, but now you have finally found the best teacher.

[@nishantchaudhary7528]

@@tathagatasharma totally agree.. 👍

[@shantinamuna-r8x]

you wrote a good poem ." you should be a poet not data scientist "....... " Bad teachers with you. you should prey them by calling all mighty " ,

[@amarchanotiya5558]

Very well structured videos you've made Sir. God Bless Teachers like you who are helping students learn better.

[@anuj_.ahlawat]

Correction :- trf1 se jo output aayega ... Vha columns k index change ho jayenge and in trf2 ... According to output from trf1 ... Sex col index = 3 and imputed_embarked col index = 1 ... So in trf2 use [1,3] instead of [1,6]

[@AmbujRai-ft5cx]

Thankyou bro....Can you please explain how and why the index of the columns are getting changed?

[@badrinath218]

Yeah I am confused about that too

[@tusharshukla9361]

what a beautiful way to explain such a complicated topic...hats off to your efforts Sir. Love allots💗💗💗

[@srihariswain2128]

Exactly

[@1234569312]

Interestingly, without Pipeline prediction is that person will survive, however with Pipeline the prediction is that the same person will NOT survive...Could you please explain this behavior??

[@anubhavgautam6586]

cause in pipeline we used feature selection step which reduced the accuracy

[@barshabanik7212]

you are making my journey a bit easier evryday .Thank you so much sir

[@ayushmeharkure5365]

sir original me hi changes krne thee n embarked ki missing values fill krdi hoti train data k or ussi data pe encode kiye hote

[@ronylpatil]

Very clear explanation. This is the first video on youtube in which each and every concept related to pipeline is explained very well. Great job sir, keep growing.

[@tusharkhatri5795]

can you please explain why he used fit transform at train_age while transform only at test at 8:22

[@darshann812]

@@tusharkhatri5795 Because if we use fit on test set then the model will also learn the test set values and then there is very high risk of overfitting.

[@ankitmishra5566]

Man , your style of delivering the content is awesome !

[@123arskas]

Thank you Nitish Singh. Your teaching style is awesome but the thing that impresses me the most is how easily you present the complex looking concepts. Your 2+ years of long journey surely paid off.

[@saumyashah6622]

Doubt : The purpose of cross validation is to train the model on "whole dataset" in "folds", then why are we passing X_train (Already splitted dataset ) for cross_val_score(). Shouldn't it be X,y instead of X_train and Y_train. Sir, please explain. Also let me know if my thinking is wrong.

[@campusx-official]

Actually, while using cross val score you pass the entire X and y

[@aination7302]

@@campusx-official How will you pass the entire dataset? The entire dataset will have missing values. And imputing missing values before the train test split leads to a data leakage problem.

[@shreyashpetkar3511]

This is my take from cross_val_score () ...when you do random test_train_split() for a dataset the algorithm would give accuracy of of model the random data which it splits as training data and test data ..but how can you say the data which was the part of the training dataset was best to train the model?... there is no way to say to model pick rows 10 to 100 for training as its best and 1 to 10 for test ..hence this things we use it cross_val_score() and we would definitely give cross_val_score() the entire independent column and dependent column data no splitting of train and test is done for this..

[@rashmik3426]

@@aination7302 We will pass entire X and y while doing the cross validation. And while doing that we pass pipe ( our pipeline consists of imputation,one,scaling,feature selection , model) , X ,Y,cv=5,scoring) ....so in this step our X and Y columns in dataset passes through the pipeline and so X and Y get through all the process . So ultimately no, matter of data leakage may come as because One of the best ways to get rid of data leakage is to perform k-fold cross validation where the overall data is divided into k parts. After dividing into k parts, we use each part as the cross-validation data and the remaining as training data. Better to use k fold cross validation or Stratifiedkfold.

[@AlayaKiDuniya]

Jb column transformer sy 3no steps ak code me kr skty tu yahan 3 bar column transformer kiu use kia hai sir?

[@dipanwitamandal7428]

He just tried to make multiple steps so that the use of pipeline is understood well( PS:this dataset is very simple)

[@poojadesai2826]

I really like your explanation, Please keep uploading more videos.

[@vanshshah6418]

Sir, You have made a very big mistake in this video. When your input passes through "trf1" the ouput indices changes such that the categorical column "Sex" and "Embarked" is now not in the same index as it was in the dataframe(in dataframe it was on 1,6) but now after "trf1" it will be on index number [1,3], so when you will check the output shape after "trf2" it will be (712, 236) it has actually encoded numerical column that is why we are getting 236 columns., But when you will actually see the output after all the transformation you will see that "sex" and "embarked" are still not encoded. The reason you are not getting error is because in third tarnsformation "trf3" you are slicing from (0,10) means selecting 10 columns out of 236 and all those 10 columns which you have sliced from 236 are in numerical form and thus you are not getting "sex" and "embarked" columns which are still not encoded and is in string format. I have copied your notebook and correctly indexced "trf2" which is [1,3] not [1,6] and just by changing this small thing my val accuracy is 80% and your's is 63%. ps: Although your concepts are very clear and its not a big deal to make such a tiny mistake considering that you have created tons of videos on different concept. Thanks a lot.

[@ananthr1583]

hi can you provide me with the code that works coz im just getting startted and im not able to figure out the cause of the error in my code.

[@akzork]

can you please share your code or notebook?

[@ele548]

^ This comment should be pinned so everyone knows about this bug. Spent a long time trying to figure out why there are 200+ columns The column transformations change the order of columns so in subsequent steps we cannot rely on the initial indices. Columns order before 'trf1' - [Pclass, Sex, Age, SibSp, Parch, Fare, Embarked]. trf1 has imputer which works on Age, Embarked - so these 2 move to front. Column order after 'trf1' - [Age, Embarked, Pclass, Sex, SibSp, Parch, Fare] - hence 'trf2' needs to be given [1,3] for OneHotEncoding

[@arpittalmale6468]

can you tell me how to check the shape after trf2 ?

[@rishibakshi2004]

Thankyou so much sir..i gained a lot...like massively a lot!!❤️❤️❤️... teacher's like you🙏🏻❤️❤️ gives us the motivation to study and explore more and more concepts ❤️

[@k-AE-MDWazidAnsari]

Bhaiyya When I am doing pipe.fit(x_train,y_train) then this error is poped up " Cannot clone object. You should provide an instance of scikit-learn estimator instead of a class."

[@prabalkuinkel4893]

There is an slight issue in the Implementation logic . Since in above pipeline output of trf1 (imputation column transformer ) will be input to trf2(one hot encoding column transformer) . Since we have imputed the age[2] and embarked[6] column first and then PASSEDTHROUGH the remaining columns . Then the output of this trf1 columns will be in the order of (columns that we have imputed and then remaining columns) i.e :(age,embarked,pclass,sex,.......and other remaining columns) .Here the index of the columns has changed from the original dataframe (df1) . In the next step result from trf1 will be passed to trf2 . And we wanted to mention column sex and embarked (index 1 and 6 respectively in the original data frame but now the index of both have been changed) , which is a mistake . So if we want to specify the correct index of column sex and embarked in trf2 it will be 1 and 3 respectively .

[@preetisrivastava1624]

why we use reshape(1,1) or reshape(1,-1)

[@RahulKumar-re4np]

sir agar main successful ho paya data science ki field mein to aapse ek baar zarur milne ki koshish karunga agar aapka response aya toh

[@ameerrace2284]

When we apply only fit in pipeline, how it will do the job of transform? For example consider scaling, we need to do both fit and transform before calling for algorithm and here nowhere transform is used, then how the values are getting transformed?

[@campusx-official]

Automatically. Sklearn is intelligent enough to do so

[@pubgdoremongamer8823]

Sir, I have a question. When you are preprocessing or modifying something in this x_train and x_test, what happens if you drop some values in these two? Will this affect the y_traina and y_test?

[@jroamindia1754]

Amazing video as well as a series. But just wanted to know as u said developer needs to write the code in production and change the code of the file **predict_without_pipeline file. But once we export our model we can handle this in our API's routes it is that hectic ?? validations laga diye to ho jaega just 1 hi bar to likhna hai code. But yes pipe mechanism is very clean and understandable

[@hamzayaseen9963]

ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 0 dimension(s). What should I do at 11:50? Can anyone help me? I can't figure it out. Thanks

[@zainfaisal3153]

I have a question We use coloumn transformer to transform/encode data and same as it is by pipeline. So, the difference is only that in column transform, we apply transformation just offline and by using pipeline, we transform our data in online mode as well Am I right? If not, please give me correct answer. Please give me reply

[@LonerFactsClub]

Sir! First half pura samajh ne ka koshish kiya....par brain mein locha hoagaya!😆....Par overall mamla samajh gaya🤌

[@IRFANSAMS]

I was lost like hell and here comes another great video with up to the core explanation.

[@PriyankBansal-d5h]

Sir i have confusion to about using of minmaxscaler transformer instead of our standardscaler . Can you give the answer ?

[@lol-ki5pd]

just a question, when we did the grid search and got our best param, do we need to change hyper parameter manually or it will assigned to my pipe object?

[@superintelligence-Supremacy]

thank you sir

[@mayankrathi6746]

i have a question as when we use column transformer and we transform the training data by applyinf different function like standard scaler , min max scaler , one hot encoder etc but we doesnt apply all these function to test data so in test data we have cat column as it is so because of this the model has to perform poor ??? plz help on this .....

[@radhikawadhawan4235]

Hi Nitish How to deal with cases when we have to write custom functions to deal with certain type of preprocessing? will that be taken care by pipeline class?

[@eeshananand3773]

Sir, I used StnadardScaler instead of MinMaxScaler but the code produces error when fitted on x_train and y_train -ValueError: Input X must be non-negative.

[@acceleratedofficial1576]

Because sir said he is using feature selection , and it doesn't take -ve value as Standardization scale value so that our mean get 0 and sd=1 so we get some -ve values whereas minmaxscaler scale value bw 0&1 so he used min mx scaler u can use standardization but then remove feature selection

[@nachoeigu]

I have a big one question: What is the difference of build a Machine Learning application with Pipeline and to build a machine learning application with a OOP technique? I see that it is the same.

[@AIWALABRO]

Really thank you so much sir, itne easy tarike se smzhane ke liye. you know, mere ek friend ne kaha tumahre pas "Ineuron" ka subscription hai phir bhi tum campusx ke videos q dekhte ho? you know what i said "great content with easiness" . that why mai campus_x ke videos dekhta hu.

[@harshkondkar3193]

Hello Sir, While using multiple column transformers in series I was facing problems with the column indices as they kept changing after each transformation. Is there a simpler way to do it or will I have to apply each transformation individually and see the column indices changes and then change my input index to the next transformer accordingly? Thanks

[@pratyushrao7979]

I'm facing the same error did you get any answers?

[@princesadariya7005]

Sir , in this pipeline it doent matter which algorithms I used , I got same accuracy for all algorithms , why?

[@kaushandutta4593]

Why do you first split the model. You can even first do the feature engineering then spilt the model

[@krishna-p9o]

X_train_transformed = np.concatenate((X_train_rem,X_train_age,X_train_sex,X_train_embarked),axis=1) X_test_transformed = np.concatenate((X_test_rem,X_test_age,X_test_sex,X_test_embarked),axis=1) ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 2 has 0 dimension(s) how to solve this error?

[@varun-hv1qi]

run all the cells again

[@PoetrybyEraj6]

Excellent code understanding and best teaching ever on KZbin✨🔥

[@pujithakantamsetti7358]

Please recommend me some KZbin channel which teach ML in English

[@EN21CSAP]

who are watching this in 2024 the sparse error will be raised it is sparse_output=' '

[@acceleratedofficial1576]

Why we did not pass scaled x_test? We were used scaled x_train so how unscaled x_test will fit in the model?

[@srikrithibhat1999]

Well explained. Nowhere I found such a great detailed explanation like this. Thank You so much.

[@rajmore4222]

Cannot clone object. You should provide an instance of scikit learn estimator instead of a class

[@chaitanyasingh6727]

sir i am working on this but i am getting error in ordinalencoding i tried my best but error couldn't solve

[@ajaykushwaha-je6mw]

Sir i need a help. Suppose I have created pipeline for missing value imputation --> onehot encoding. so can i use it for both training and testing data ?

[@shubhamshakya-w9w]

why the training data gets fixed and the test data does not in simpleinputer

[@aksmalviyan8342]

in X_train_sex you passed array i.e X_train[['sex']] in ohe but in X_train_embarked you passed just X_train_embarked varaiable while doing ohe, why? Plz xplain

[@smile93]

I am also confused there

[@MuhammadJunaid-yr8jd]

I really like your explanation, Please keep uploading more videos.

[@utkarshchalsey241]

I have done the exect as you have done. But I am getting an error ValueError: could not convert string to float: 'male'

[@aqilsaboor9988]

Hi Nitish sir! Hope you are doing well Nitish sir. First of all, Thank you so much for all the knowledge that you are sharing with us free on KZbin. Sir, I am a college student and tomorrow is my AI & ML engineer interview at a big tech company. I want to revise all the stuff that I learned for your KZbin channel playlist 100 Days of Machine Learning. Can you please share with me your Microsoft One Note file which you used in your playlist? So I can revise all the concepts of ML in a few days for my first AI & ML interview. Again Thank you so much for all of your efforts which you give us on KZbin without any fee. Kind Regards Aqil Saboor

[@tejasmaheshkalebt22m011]

whats the update of your interview ?

[@katadermaro]

I have a question if anyone can answer, that'd be great. Thanks! I have noticed that when using ColumnTransformer, the values get encoded and it comes at the beginning of the output array. Suppose I am encoding the [-1] column, it will still come at the beginning. So, when using pipelines, if I use 3 encoding using 1 column transformer, will that be a problem? Or does all the encoders get fitted once and then transformed, in that case it might not be a problem.

[@Nudaykumar]

Yes, i faced this issue. It made me to change the column number in columnTransformer bye executing each transformer in pipeline. Not sure how to make the column sequence in stable.

[@AhmarDataScientist]

How can I access your notes(handwritten and drawing),

[@sagarkanake3357]

Sir, these videos are really helping me to understand ML concepts. Thank you so much, sir.

[@sandeep50378]

Why we have not apply Standardisation on Age and Fare ?

[@shivangitripathi2454]

Thanks a lot for this playlist. You have literally explained it so well and especially from the real project perspective. Thanks.

[@mohindergupta6305]

Your videos are excellent . Very easy to understand. Do you have a video on deployment of model to AWS EC2 ?

[@campusx-official]

kzbin.info/www/bejne/laPaf4eParKhapI

[@HimanshuSharma-we5li]

Aap pehle mil gaye hote to....mera itna time waste na hota🙄🙄🙄🙄

[@anubhavgautam6586]

Thank you sir for this amazing lecture

[@saumyashah6622]

Sir, if I am making pipeline for polynomial regression and then making model for hyperparameter tuning, Is this the correct code? pr1 = PolynomialFeatures() pr2 = LinearRegression() pipe_pr = Pipeline([ ('pr1', pr1), ('pr2', pr2) ]) params_pr = { pr1__degree : [2,3,4,5,6,7,8] } grid = GridSearchCV(pipe_pr, params_pr,cv=10, scoring = 'accuracy') grid.fit(X_train,y_train)

[@campusx-official]

Seems right

[@saumyashah6622]

@@campusx-official thanks 😊

[@PriyankBansal-d5h]

This channel is awesome no one can bit of this man

[@lukealadeen7836]

I wish this was in English.... looked like a good video

[@joydeepmondal320]

At least the guy survived without pipeline🥲

[@nainarahangdale1261]

how did uh add the mean ? does simple imputer acts as a mean?

[@ecanalysis7589]

sir ci/cd pipeline pe tutorial laiye🙏🙏🙏🙏

[@drip_]

33:24 what is the reason behind this?

[@ghostofuchiha8124]

But how will it handle droping columns ; Theres nothing in pipeline where it drops useless columns automatically , as during testing here we are only providing required values in test not all the columns as present in original dataset.

[@aradhanarana9592]

Hats off to you sir, the efforts and honesty you put into making any video to make everyone understand the concept in depth, are remarkable. This is what we call hard-earned success with full of honesty and dedication. This channel should reach a great height and will soon become one of the best channels for data science courses. Thanks and keep up the great work.🙌

[@salmanali-xx1cg]

after this step : pipe.predict(test_input2) I get the following error AttributeError: 'OneHotEncoder' object has no attribute '_infrequent_enabled' plz help

[@SaicharanErram]

ab toh mai kyaa karunga concatinate karunga ," Kyoh ki mai gareeb huuu" , kuch nahi kar saktha ....😂😂😂 i was listening the class with intensity you made me laugh many times. Im in day 29. On my 100 th video ill do my guru dakshina for sure . huge respect to you guru ji

[@maitrijain7758]

Ran out of input error how to solve this erroe

[@NikunjBosamiya-hb8pf]

I am getting error msg while fitting the pipe. The error msg is could not convert string to float 'S'. I have done everything same as shown in the video. Please help me solving this error.

[@shankarpendse]

This is because, columnTransformer does not preserve the column order as in pandas dataframe, so which ever column you transform first, that will be the first column in the resulting numpy array. I am soon uploading a video on this. The solution is bruteforce, where we have to provide the order of the columns in every step. @CampusX Nitish Singh, any solution for this?

[@shankarpendse]

One solution which I found out is to specify the columns in oder and call it as a "passthrough" column, then specify your columns on which you want to apply the transormation. Something like this: # handle missing values: ctrf_impute = ColumnTransformer([ ("pclass_sex", "passthrough", [0,1]), ("impute_age", SimpleImputer(), [2]), ("sibsp_parch_fare", "passthrough", [3,4,5]), ("impute embarked", SimpleImputer(strategy = 'most_frequent'), [6]) ], verbose = True, remainder='passthrough')

[@jpsama7817]

Nitish Sir in this video, while using transformers for the same test input we got result as 1 in pipeline and 0 in normal way. two diff outputs, iam not knowing which one to interpret as correct. Hope you answer this.

[@Faisal-ww4oj]

kisi k pass data pipeline ki ppt file ha ??

[@tusharkhatri5795]

can any one explain why he used fit transform at train_age while transform only at test at 8:22

[@apratimmehta1828]

Fit transform is used to transform and fit train data. Test data is only transformed and not fit to avoid info leakage.

[@anuragshrivastava7855]

one of the best data science channel