Function Transformer | Log Transform | Reciprocal Transform | Square Root Transform

[@webtechbysuraj5942]

26:38 --I just wrote the simple code instead of using column transformer and to use function transformer on the Fare column. accuracy was improved as you said. Thanks

[@tarunmohapatra5734]

I think there is a mistake while applying cross-validation on entire datasets because ideally it should be applied to training data. This is to prevent information leakage from the test set into the training process. The purpose of the test set is to simulate unseen data and evaluate the final model's performance. If you perform cross-validation before the train-test split, you may inadvertently use information from the test set leading to an overly optimistic assessment of the model's performance

[@narendraparmar1631]

Marvelous knowledge Thanks sir for your efforts😀

[@jatinnandwani6678]

This was mindblowingly awesome! Thanks Nitish

[@ug1880]

Even if we use normal method without pipeline, we need not to change the predict function, ryte ? We just change pkl file....

[@acharjyaarijit]

Sir, u r the best.. u know very think..

[@dostirathi8254]

i want to ask that is cross-validation used over training data or can apply over the whole x and y

[@shantinamuna-r8x]

so handling skewed or non-linear data mean scaling the data and feed it to the model directly . in addition it is use in EDA or feature engineering and moreover can we use standard scaler or any other scaler for data handling (skewed or non-linear)

[@parikshitshahane6799]

One correction: I think square root transformation works good on left skewed data, not square transformation

[@mukteshsingh8370]

Day - 30

[@akshaychauhan545]

Should we remove outliers first or we can remove outliers after using transformer

[@rajsharma-bd3sl]

When we are applying functional transformer to Age column of the training data , it is getting left skewed and not normally distributed. Is it the same for anyone ?

[@jamalshah3657]

can you provide us the OneNote Notes for 100 days of ML?

[@rehanansari8154]

I think nAge column was right skewed than the fare column

[@guljitsodhi6149]

HI Sir, i started watching your vedio's ,very informative,but i have some errors while using some other similar data, can you help explaining and correcting my doubts?, i woulld really appreciate.Thanks

[@sandipansarkar9211]

finished watching and coding

[@debjeetmukherjee4591]

Kaha impovement hua cross validation k baad Sir ji samjha nehi

[@dostirathi8254]

if there is fillna function then why we use simpleimputer function over train and test data

[@rohitvishwakarma2871]

even i want to know this

[@1981Praveer]

#campux @18:08 min, why did we use mean value. it might inject outliers. I think the median should be better. what's your opinion

[@shubhamgosain4534]

Thank you sir

[@murumathi4307]

This transform control than outlayer sir

[@vishalodyssey]

IF MY DATASET HAS 10 COLUMN AND FEW OF THEM ARE LEFT SKEWED AND FEW ARE RIGHT SKEWED AND SOME ARE NORMALLY DISTRIBUTED HOW DO WE HANDLE THOSE , DO WE APPLY DIFFERENT DIFFERENT TRANSFORMATION ON THE BASIS OF COLUMN

[@abhisheksen4440]

why we do log transformation when we have Standardization ?

[@rkpatel-r5q]

why without using ML pipeline output [survived=arr[1]] and output of with using ML pipeline [notsurvived=arr[0] are not same ?

[@mayurnaktode592]

@Campusx sir if data is skew or not normally distributed it means we have a outliers correct? For removing outliers we use trimming or capping. And if we use log transform then is it like it will adjust the outliers and convert it into normally distributed?

[@agnimitram340]

I don't think non normal data means presence of outlier. Like Binomial distribution, Chi square distribution are not normal does this mean they have outliers??

[@aaditstudent]

even after a converting to a normal distribution you can get rid of the outliers by removing from the 3rd or 4th std deviation

[@theanalyst9629]

Thanks man

[@muhammadmustafa3158]

thanks sir !!

[@mansichougule1174]

I'm not getting the coding part, can anyone please suggest me something??

[@malikahsan6648]

amazing

[@gauravgupta4983]

Sir log, sd and normalization tenio ka use same hi hota hai kiya? Teenon concept mein ham data ko ek normal range mein lekar aate hai, I am right

[@ParthivShah]

Thank You.

[@amritajoshi8729]

sir cross validation aapne entire data me ku laga diya. aise to before and after results sahi nahi aayenge ... :)

[@sarbajitde2547]

Sir, we can use the central limit theorem then why do we use such transformations to convert a pdf to the normal distribution?

[@yashjain6372]

doubt : Sir LR assume residual to be Normally distributed not the data. So, why we suddenly start making features to be ND . if Residual of LR is not normally ditributed that means relationship is not liner and we try to make those column ND which are not linear with Target Variable, This is my understand . Please explain.

[@Ganeshkakade454]

Hi yash...Nice Question...I had also same question..Yeah we know there is assumption that residual should be normallly dirstibuted but also when u get data as normally distributed in certain algos like LR, logR then model performance gets better as we can reduce heterosedasticity from model...when u data is normally distruibuted model statistial power of compuattation gets increase..hope u got u r answer,,if U knew anything more ..plz feel free to share

[@bepositivefoxx2241]

Sir ye normalisation bhi to same kam krta hai

[@beautyisinmind2163]

normalization does not guarantee normal

[@excalibur2889]

function transformer come under which part of feature engineering ?

[@beautyisinmind2163]

To make features normally distributed

[@parthraghuwanshi2929]

@@beautyisinmind2163 does it is method of handling outlier or outlier should be handle differently

[@beautyisinmind2163]

@@parthraghuwanshi2929 For handling outlier you can use other method, transformation is especially to make feature normal for linear model like LR, LogR, NB etc.

[@MRAgundli]

done

[@yujiitadori225]

i badly need this answer i have looked few place but i ain't getting the correct answer! Variable transformation( Function and Power Transform ) considered as featured transformation technique or this is another part of feature engineering method but not included in feature transformation ?

[@depalvveturkar]

Hello Sir, After applying Function Transformer my X_train shape is changing from (719,2) to (80,7) can you help me why is this happening, my X_test shape is intact. I am following your GitHub but still facing issues. Please help

[@depalvveturkar]

It is resolved

[@ajaykushwaha-je6mw]

I have a doubt in data preprocessing. First we remove outlier --> Feature scaling --> Gaussian Distribution or remove outlier --> Gaussian Distribution --> Feature scaling kindly help ?

[@viratmani7011]

Second one

[@surajmeena6797]

feature scaling shall be applied at last

[@as8401]

this project difficult to understand