Parent Child design pattern with Parameters and Expressions in Azure Data Factory

In this video we look at developing a parent child design pattern in Azure Data Factory with Parameters and Expressions. In order to pull this off we also use the Lookup, For Each, and Execute Pipeline activities!
I've done previous videos that are more specific on individual topics, you can find those below:
Working with Expression:
• Azure Data Factory - D...
Filter and Foreach activities:
• Azure Data Factory - F...
LOOKUP AND IF Condition activities:
• Azure Data Factory - L...
#MitchellPearson #MitchellSQL

[@tenghover]

High quality video, as a not native English user, it's easy to get understanding with ur speaking speed and pace. Please keep making all possible design scenarios about ADF. Thank you !!

[@vikashkeshari6258]

Simple and The Best Video !!! Thank you Mitchell, You are awesome...

[@soucianceeqdamrashti8175]

Great pedagogical video Mitchell! Excellent series! One tip for a new video, maybe go through some best practises and antipatterns for ADF..

[@anuradha8700]

Very well explained.Thanks for all your effort

[@MitchellPearson]

Thank you Anuradha!

[@samanthahughes933]

Thank you so much! This was incredibly helpful.

[@ABBORStern]

How could we dynamically change also the columns names in such a multi query . like for example, change empty spaces to underscores ?

[@alessiomartello2692]

Great video, solved a problem I was having. Thanks a lot

[@MitchellPearson]

Great to hear!

[@lucassage8820]

A trick: watch series on Kaldrostream. Been using them for watching all kinds of movies these days.

[@kasenalfredo2396]

@Lucas Sage definitely, have been watching on KaldroStream for months myself :)

[@soucianceeqdamrashti8175]

Do you really need the curly bracket to refer to the item properties? It seems to work without it as well or?

[@priyankapatil9930]

Hi Mitchell, Your video has helped me a lot...Could you please make video for the same but in vice versa like azure data lake to azure sql...

[@muhammadrahman3510]

Hi Mitchell, Thanks so much for another great video done in very professional way. I had a question though: We used to change the size(height and width) of individual SSIS components in the Package so that the longer description/label in the component can be seen and can be arranged properly. How can we do that same thing for an Activity in ADF Pipeline? Your response would be highly appreciated.

[@CarlHunter88]

Hey Mitchell, great video - how would you handle the above scenario if some of the tables needed an incremental/delta load with an upsert i.e. insert new and changed rows?

[@ajaxaj4885]

Why did you use child pipeline inside. We can also use copy activity directly right?

[@dperezc88]

Hello, Mitchell. Thank you for your videos; this one was really helpful. I'd like to ingest data into my data lake in a dynamic method, similar to this video, except I'd like to dynamically construct sub folders like this: \Raw\DataSource\Entity\YYYY\MM\DD of ingestion\ File.extension Do you have any suggestions for how I can set up the YYYY/MM/DD folders?

[@NeumsFor9]

You should cover custom eve't triggers.......

[@shruthil7913]

Hi Mitchell, Thank you for all the videos you have uploaded. I am working on the project and below is my scenario. I have four excel/csv files - File1, File2,File3 and File4 with different column names Tables in Azure SQL -> 4 staging table, 4 master table and I have created 4 stored procedure in SQL (update file on daily basis) respectively for each file. I get 4 files with the same file names to storage blob (all 4 files in one container) on daily basis, I need to load the files from blob to Azure SQL. Let me know if you can able to upload video similar to above scenario.

[@MitchellPearson]

You can definitely handle this pretty easily in ADF because of the ability to easily deal with late arriving schemas. You could either create four separate pipelines and eight datasets (1 for each source and 1 for each sink). Or you can go the approach of this video and try to make it all dynamic. I do think this would make a good KZbin video :) Unfortunately, it would be a few weeks before I got around to recording a video for this!

[@tarun007]

It was Fantastic..

[@HarishKumar-sn8ph]

Can we do this just reverse like DynamicFile_timestamp.csv to sql table.

[@Reddy23295]

Vgud explaination