Renamed columns or removed the spaces and then created

Hi actually that concept covered in the Data validation playlist. By creating metadata files. Thanks

@@DataSpark45 can you share you email id i want to communicate with you

def SchemaComparision(controldf, spsession, refdf): try: #iterate controldf and get the filename and filepath for x in controldf.collect(): filename = x['filename'] #print(filename) filepath = x['filepath'] #print(filepath) #define the dataframes from the filepaths print("Data frame is creating for {} or {}".format(filepath, filename)) dfs = spsession.read.format('csv').option('header', True).option('inferSchema', True).load(filepath) print("DF Created for {} or {}".format(filepath, filename)) ref_filter = refdf.filter(col('SrcFileName') == filename) for x in ref_filter.collect(): columnNames = x['SrcColumns'] refTypes = x['SrcColumnType'] #print(columnNames) columnNamesList = [x.strip().lower() for x in columnNames.split(",")] refTypesList = [x.strip().lower() for x in refTypes.split(",")] #print(refTypesList) dfsTypes = dfs.schema[columnNames].dataType.simpleString() #StringType() : string , IntergerType() : int dfsTypesList = [x.strip().lower() for x in dfsTypes.split(",")] # columnName : Row id, DataFrameType : int, reftype: int missmatchedcolumns = [(col_name, df_types, ref_types) for (col_name, df_types, ref_types) in zip(columnNamesList, dfsTypesList, refTypesList) if dfsTypesList != refTypesList] if missmatchedcolumns : print("schema comparision has been failed or missmatched for this {}".format(filename)) for col_name, df_types, ref_types in missmatchedcolumns: print(f"columnName : {col_name}, DataFrameType : {df_types}, referenceType : {ref_types}") else: print("Schema comaprision is done and success for {}".format(filename)) except Exception as e: print("An error occured : ", str(e)) return False

Thanks a million bro

Hi, can you please share the details or pic where u have the doubt

Glad to hear that!

Sorry to say this bro , unfortunately we lost those files

Thanks a million bro

Hi, u can contact me through LinkedIn Lokeswar Reddy Valluru

Set values will come from reference df .so it always a constant one

Hi in the description i provided the link bro

Thanks for asking, will do that one for sure .

Glad to hear it!

Could you please upload a video on differences between Data Lake vs Data warehouse vs Delta Tables

Thanks a million. That will do for sure

Glad to hear that

thank you bro

Thank you, I will

Glad it helped!

Glad to hear that

Hi, You can use databricks, then you have to play around dbutils.fs methods in order to get the list / file path as we did in get_env.py file. Thank you

unfortunately, we lost those files and account

Hi, r u talking about source parquet file! It's under source folder

No need brother, we can continue with out defining spark session, i just kept for practice

yes, test.py here i used for just to run the functions and about udfs.py please watch it from 15:oo min onwards

Thank ypu bro

you can reach me on linkedin id : valluru lokeswar reddy

Take a look at this : drive.google.com/drive/folders/1XMthOh9IVAScA8Lk-wfbBnKCEtmZ6UKF?usp=sharing

Sure I will

we defined as a table in any DataBase. As of know i used them as a csv

sure, here is the link drive.google.com/drive/folders/1QD8635pBSzDtxI-ykTx8yquop2i4Xghn?usp=sharing

Thanks@@DataSpark45

did your problem resolved? @@vishavsi

Hi when you write df to hive generally we use df.saveasTable() . so that table will created in Hive environment then we can use spark.sql(select * from table). If you don't want to use HIVE then probably we use df.registerTempTable("TableName")

where did you got this error bro

@@DataSpark45 while downloading data. But i got data from part 2

Thank you, I will

As of Now i'm not providing training bro.But you can reach out me at any time for any sort of doubts Thank you

Certainly, whatever the order will mention at reference_df is the correct order as we expect.If we sort dfs column names in alphabetical order then their would be chances of failure. Thank you