16. Validate file schema before processing in Azure Data Factory

In this video, I discussed about Validate file schema before processing in Azure Data Factory
Link for Python Playlist:
• 1. Introduction to Python
Link for Azure Databricks Play list:
• 1. Introduction to Az...
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
Link for Azure Data Factory Real time Scenarios
• 1. Handle Error Rows i...
Link for Azure LogicApps playlist
• 1. Introduction to Azu...
#Azure #AzureDataFactory #DataFactory

[@shaileshsondawale8765]

You are amazing Maheer... Best in Class...

[@tanushreenagar3116]

Too nice all I watched whole playlist

[@WafaStudies]

Thank you 😊. I know you. Many comments from you on my videos. Thanks. Kindly reach on [e-mail] for any queries🙂🙏

[@tanushreenagar3116]

@@WafaStudies yes please I want your help

[@WafaStudies]

@@tanushreenagar3116 sure. Kindly mail me details. Will try our best to help

[@tanushreenagar3116]

@@WafaStudies thanks

[@balamanikanta7685]

Nice Video.. How can we do this with multiple files having multiple schemas sir... Any hint?

[@shishirranjan4961]

did you get answer or any suggestion. I have same scenario

[@anveshreddy2655]

Nice explanation... please can you make video on SCD type 2 table creation with start date and end date and active flag in ADF ?

[@suvratrai8873]

Thanks for this video. Will it give error if the order of columns is different?

[@rameshkumar-dw8he]

good explanation i need to validate csv file columns and datatype and size and data if its failed then need to trigger mail can you post any videos related to this

[@basantdwivedi4171]

Can you tell me logic, there are 20 files in the directory and I have to check the schema of each with a reference schema. In case of mismatch in the schema that file should be moved to some other directory and in case of matching schema do copy activity in adf. If possible make a video otherwise just reply.

[@srinubathina7191]

Thank You Sir

[@AnkitSingh-tz3ix]

i am new to adf m inserting data into one of the table using hdi sight and then once it is inserted want to know the count basically no of records inserted .how we can do it please suggest?

[@sid5201]

GetmetaData does not work in scenarios where you are dealing with very large files as it has size limitation of 4 MB.

[@azurespace]

It looks like the size limitation is of metadata, not the actual size of the file. As per docs, they say this "The maximum size of returned metadata is 4 MB."

[@user-xx5xk1wv1z]

My csv file is having product id and the target table is having product_id as int datatype. Now adf is thinking csv file product_id as string. Now the pipeline is getting failed in copy activity. Anybody help me how to do this?

[@sujitjujgar2982]

What if we don't have any file in blob storage

[@WafaStudies]

Then validation activity keep try to check fail until it reachs timeout

[@Poori1810]

how to compare the schema of a file and table?

[@msairakesh]

Thanks for your support............ Need your support for data engineer resumes could you please help on this? thanks in advance.

[@sehajshangari5707]

WHY CANT WE JUST USE VALIDATE SCHEMA FOR THIS?

[@frankopokukoduah194]

Yes I agree

[@BannedUser5678]

I'm not getting structure option in fieldlist in get meta data 2 what should we do ??@WafaStudies