How to read and write multiple csv files from different formats using Databricks

Hi All,
in this video i have covered how to read and write multiple cvs files from different files format present in ADLS into seperate datframes,
This was question asked in one of Wallmart Pyspark Interview
Source code Github:
github.com/Anu...
#azuredatabricks #pyspark #databricks #python #dataengineering #azuredataengineer

[@rahuldave6699]

i was asked this question in TCS interview, but some changes are there as that we have files in source with different-different schemas in source and we have to write the data into the sink with different table with the respective matching schema dynamically

[@thecloudbox]

Great , I hope you were able to answer that

[@rahuldave6699]

@@thecloudbox not able to answer how to match schema of tables for that for that particular dataframe to put dynamically not manually to insert data to table

[@swapnilnarvekar3139]

Hi , Will the dataframe size matter when storing it into the dictionary?

[@thecloudbox]

It is taking the data and writing ideally it should not matter, I checked with more than 1 gb of files it is working

[@naveenreddybedadala]

Can't we rename the writing file name with the actual CSV name , because u can see the CSV file is named with some random name

[@thecloudbox]

Please refer the another video in playlist I have shown how to do that