Data Validation Using Pyspark || ColumnPositionComparision ||

How we can develop a function or script using Pyspark to compare ColumnPosition while dumping data into raw layer from stage or source layer.
#pyspark
#databricks
#dataanalytics
#spark
#interviewquestions
#pythonprogramming
#dataengineering
linkedin :
/ lokeswar-reddy-valluru...

[@MuzicForSoul]

why we have to do ColumnPositionComparision? shouldn't the column name comparison you did earlier catch this?

[@vamshimerugu6184]

Sir Can you make a video on how to connect adls to DataBricks using Service principle

[@DataSpark45]

Thanks for asking, will do that one for sure .

[@DuyTran-tx5jq]

Can you do end to end portfolio project please

[@DataSpark45]

We don't have the azure account bro as of Now. Once after creating the account we will do it for sure. Thank you

[@DuyTran-tx5jq]

@@DataSpark45 sure! Btw love your content so much

[@vinothkannaramsingh8224]

Sort the both ref/df column name based on alphabetical order and compare column names ? will it be sufficient ?

[@DataSpark45]

Certainly, whatever the order will mention at reference_df is the correct order as we expect.If we sort dfs column names in alphabetical order then their would be chances of failure. Thank you

[@MuzicForSoul]

sir, can you please also show us the run failing, you are only showing passing case, when I tested by swaping the columns in dataframe it is still not failing because the set still have them in same order.

[@DataSpark45]

Set values will come from reference df .so it always a constant one