Python Feature Selection: Remove Multicollinearity from Machine Learning Model in Python

[@Nomar_7]

nice video, what about checking from where the high threshold is coming and comparing the correlation with the target column and only dropping the one with the less correlation

[@StatsWire]

We can decide the threshold and see which columns are having high correlation

[@oomraden]

Hi thanks for the video! Would'nt that remove all high correlated columns instead of just leaving one column for every relationship?

[@StatsWire]

It will leave one column for every relationship.

[@SaifTreks]

@@StatsWire Great video! I don't quite get what you mean here. isn't the list returning every column that has a high correlation based on threshold? And then we're proceeding to remove all those columns. Should we intentionally just keep one instead of removing all of them. How is it automatically keeping one if that's what you are saying.

[@anishdeshpande395]

Is this method better than variance inflation factor?

[@StatsWire]

Both of them are good.

[@baburamchaudhary159]

I have been following you for feature selection, covered forward, backward, exhaustive, variance threshold, chi2, etc. You have not shared the dataset, in them. for us to follow along you, why don't you share dataset?

[@StatsWire]

Please find the dataset link: github.com/siddiquiamir/Data

[@d1pranjal]

How are diagonal elements being handled in the user defined function correlation(df, threshold) ?

[@StatsWire]

I did not get your question

[@d1pranjal]

@@StatsWire at diagonal elements... value is 1 > threshold... so all elements will show up in output

[@StatsWire]

@@d1pranjal Ok, hope you found the solution yourself.

[@AnanyaJoshi-g2x]

what about a scenario where the order of the columns change? since we're checking for adjacent columns and their correlations to be more than the threshold and then remove the first out of the two in case the threshold is matched or passed, if I change the order of columns, the result received will be different. is that going to a correct list of features as well?

[@StatsWire]

That is completely ok. You can change the order.

[@AnanyaJoshi-g2x]

@@StatsWire thanks for the reply, I did change the order and got a different set of features. Built an XgBoost model with both sets of features and got extremely different forecasts and accuracies in both cases. How do I decide which is correct?

[@StatsWire]

Yes, that is going to be a correct feature list. You can change column positions no problem at all.

[@protapmaitra5049]

This video was really helpful, thanks a ton.

[@StatsWire]

You're welcome

[@michaelsagols8295]

Thank you for the video! very well explained! keep it up!

[@StatsWire]

Thank you for your kind words.

[@jorge1869]

Excellent, thank you very much.

[@StatsWire]

I'm glad you liked it. You're welcome

[@akiwhitesoyo918]

Nice ! Would it be the same if we use PCA to avoid multicollinearity ?

[@StatsWire]

Thank you! There would be some minor differences.

[@mazharalamsiddiqui6904]

Very nice

[@StatsWire]

Thank you

[@farahamirah2091]

hi, how to get this dataset?

[@StatsWire]

Hi, please find the dataset: github.com/siddiquiamir/Feature-Selection

[@farahamirah2091]

@@StatsWire thank you

[@StatsWire]

@@farahamirah2091 You're welcome!

[@maskman9630]

how to find collinearity for categorical features

[@StatsWire]

You can use chi-square

[@maskman9630]

@@StatsWire thanks brother...... Suppose I have done chi2 test of independent variable and dependent variables.and then i got f and p values, then how can I select features based on those f and p values...? Will u please clarify this brother

[@StatsWire]

@@maskman9630 Select the variable whose p value is less compared to other variables.

[@gisflow406]

This wasn't helpful at all. You just picked one of the correlated variables randomly without additional criteria. Anyways, correlation matrix can't do much. It's much more reliable to use VIF or hierarchical clustering for feature selection.

[@StatsWire]

Hi, this is for demonstration purposes. You can deep dive and pick the variables based on your selection criteria:)

[@naveedullah390]

when i enter the code line =====> corrmatrix = X_train.corr() it gives the error of =========> AttributeError: 'numpy.ndarray' object has no attribute 'corr'

[@StatsWire]

You need to make sure your data is in correct format.