#51. Azure Data Factory - Use Event triggers to copy files coming in different times

[@rambaburams443]

Thank you So much for your efforts, time and detail explanations on each topic.

[@AllAboutBI]

Glad you like them!

[@AKHILESHKUMAR-sz7il]

can you upload video on Incremental loading of multiple tables using ADF lookup activity. Thank you.

[@trishachatterjee1126]

Your videos are so nicely explained. Thanks a lot for explaining stuff using various use cases. It is so very useful. So, regarding this session, I have a question. If there are multiple files coming in the same input folder from different sources and that need to be separated dynamically to different containers(different sink locations)according to the file names, do event trigger can be implemented there? Thank you in advance.

[@ketanmehta3058]

Is it possible to define source directory name dynamically @All About BI ?

[@ketanmehta3058]

Ex: trigger will keep polling different directories to check file availablity in year/month/day/myfile.txt directory

[@AmitKumar-es5ns]

Thanks Mam, Que: I want it to run the pipeline using event based trigger when total fives files comes in input folder. how to do?

[@vijaybodkhe8379]

Thanks for sharing.

[@babuganesh2000]

Very good explaination

[@AllAboutBI]

Thanks so much!

[@tjsr4867]

Super. Explained easily

[@raghunandan3068]

Thanks for explanation! on copying data from folder1 to folder2. Is there a ways we move the file from folder2 (with some transformation on the file) to folder3 in the same pipeline.

[@AllAboutBI]

definitely possible. we can implement any logic inside pipeline

[@raghunandan3068]

Thankyou very much

[@sampathkumar7397]

Thank you for the details and nice video. Similarly, how do we create Event trigger for Data Flow?

[@AllAboutBI]

Triggers are generic to pipelines.. it can have data flows, copy activity or any other activity. Is your question different ? Pls let me knlw

[@sampathkumar7397]

@@AllAboutBI Thank you, you have answered my question.

[@DeviMahendra]

Hi Ma'am, please help me in clarifying this doubt. There is a event based trigger for the blob storage, once triggered the pipeline executes a stored procedure. It works fine if we get 5 or 10 files at a time. But there are times, about 160 files come to blob at one shot. With constraint of having only 100 queries in queue in adf and concurrency of 50 parallel running pipelines, remaining files are getting failed. It will be really helpful if you can give your suggestions. Thank you.

[@AllAboutBI]

Hi request you to post your question on Microsoft Q&A platform for immediate support

[@gagandeepkalra3775]

Hi Ma'am, I am using same event based trigger to execute 2 pipelines automatically. I get 2 source files (.csv) in my storage account but both my pipelines are executing multiple times instead of executing 1 time each. Please help me out in understanding this scenario.

[@AllAboutBI]

It's by design, when a storage event trigger is created, it runs whenever the matched file or pattern is found in the given folder. Alternatively, you can archive the processed file, so the same file is not processed multiple times.

[@omprakash-bd6tq]

Hi madam, I have a scenario when 2 files r uploaded to blob using event trigger can I needed transfer the 2 files different path can we do it..??

[@AllAboutBI]

Yes, use copy activity... To move to different container or data lake

[@omprakash-bd6tq]

@@AllAboutBI thanks so much for replying but destination is data warehouse 2 tables, Ya using copy activity we can do but destination want to give only 1 but I need give 2 tables (X. Txt ->A table,Y.txt->B table)

[@raghunandan3068]

using Event based trigger