Azure Data Factory Parametrization Tutorial

[@raviv5109]

Are you real? You really explain and show demo in such a flawless manner. Thanks you so much, your Azure videos are happy at every step.

[@AdamMarczakYT]

Thank you so much :)

[@nicholasgregory4774]

Another great one Adam, consider me a new student! Short enough in length while fitting all the relevant information in + easy to digest, without overly complicating things. Your PowerPoints with your explanations are second to none.

[@AdamMarczakYT]

Awesome, thank you!

[@eerosiljander4622]

Thanks Adam for these hands on videos. Best wishes,ES, Data engineer & Scientist.

[@carl33p]

Demos are the best way to learn, and your demo is as good as it gets. Build, run, build, run- the incremental style allows viewer to understand slowly.

[@AdamMarczakYT]

Couldn't agree more! Thanks!

[@nikhilpatil2021]

Thank you Adam, this is the best tutorial on Azure Data Factory I have seen! I am definitely going to try these demos. Thank you!

[@AdamMarczakYT]

Great to hear! Cheers!

[@rajdeepmandal1470]

The quarantine days wouldn't be this useful without your teaching :) thank you Adam. You are the best!!!

[@AdamMarczakYT]

Wow, thank you! :)

[@nupoorsingh3323]

It's great support for people like me who do not have a mentor. Thanks for uploading these videos and helping me to understand the workflow and survive.

[@ghay3]

My goto KZbin channel for anything Azure - you are doing the greatest service by sharing your knowledge and responding to all the queries! many thanks, Adam. Hope you can also enlighten the community on AWS :)

[@AdamMarczakYT]

Wow, thank you! glad to help!

[@tatianebrandao5487]

I'm just starting with Synapse/DF and I found your video extremely helpful for beginners. I managed to go through your video, complete the exercise on my own and everything made so much sense! Your way of explaining is very clear! Thank you very much for creating the video :)

[@sankaji20]

You are doing great job ADAM, 20-30 minutes videos of your's is equal to 60minuts of others. And quality of your vdo is un match-able. Now I am GCP guy but also very much interested in learning azure from your videos only.. keep uploading azure videos. Thanks so much.. doing wonderful job for learners.. you will get a lot of blessings.. Thumbs-up for you..

[@AdamMarczakYT]

Wow, thanks! Appreciated!

[@sraoarjun]

One word - FANTASTIC !! ... Thank you Adam , keep up the good work in educating several people .Also request you to put some videos on Synapse analytics ...

[@AdamMarczakYT]

Thanks! Will do!

[@mayur6816]

Thanks, Adam. This is the best tutorial on ADF.

[@AdamMarczakYT]

Glad you think so! Thanks!

[@mashagalitskaia8642]

Thanks a ton for this video! It helped me a lot. Very precise and detailed, I had no questions left:)

[@dabay200]

This is a really amazing video, so much better than Microsoft's tutorials.

[@AdamMarczakYT]

Thanks again Dinal. You are too kind :).

[@johnd3601]

Thanks this is a big help, its amazing how much you don't know you know. This has opened up a lot of possible automation and saved me loads of time once I manage to apply this to my situation.

[@TeymurRzayev]

Actually liked it before watching the video ;) Brilliant content, perfect explanation!

[@barryjohnson8915]

Really excellent tutorials. Very simply explained, easy to follow, fantastic stuff.

[@AdamMarczakYT]

Thanks Barry :) Glad to have you here.

[@eatingnetwork6474]

great video, once again very clear and concise, thanks Adam

[@AdamMarczakYT]

Thanks 👍

[@arnabmondal5596]

Awesome clip. Currently, better content than many available now.

[@AdamMarczakYT]

Thank you so much :) glad that you enjoyed it.

[@srinivasreddysankasani5140]

Thank you Adam for your videos. your explanation is really awesome. looking for more videos

[@AdamMarczakYT]

Glad you like them!

[@jagadishparale8632]

Hey Adam,It was excellent tutorial.I would like you to do more videos on deployment of Azure Data Factory pipelines when we use parameters

[@AdamMarczakYT]

Thanks! You are in luck, new video tomorrow is also on ADF, stay tuned!

[@snmailist1470]

Thank you Adam. I'm very appreciate your effort 👍👍👍👍 You give me a description what's going on the cloud.

[@AdamMarczakYT]

Glad to hear it , thank you!

[@sazidaanjumsheikh6817]

Thank you Adam, in a very short video you made me understand the ADF with a demo instead of Theory.. Nice content... :-)

[@iChrisBirch]

Great tutorial, answered my questions as I had them, great logical flow.

[@santanughosal9785]

Awesome, simple and hands on video to understand the concept. Thank you

[@lehai9172]

Fantastic tutorials, I've learned a lot from your videos. Thank you for sharing your experiences, Adam.

[@AdamMarczakYT]

My pleasure!

[@baladenmark]

Wow. Thank you Adam. I really can't thank you enough.

[@AdamMarczakYT]

Glad I could help!

[@ashishjuneja6574]

Super awesome !! Great going Adam.

[@igiq]

Keep it up Adam, your videos are very helpful !

[@AdamMarczakYT]

Thanks! will do!

[@joshiabhinav]

Man you are awesome. you present so well and explain so well. Love the way you teach !

[@AdamMarczakYT]

I appreciate that!

[@SuperSanjays]

Great information you have shared Adam. It helps me a lot.

[@venkataramakrishnamagapu7645]

Excellent content. Simple and informative.

[@AdamMarczakYT]

Much appreciated!

[@IntelliCloudTech]

Clearly explained at every point. thanks for the video.

[@pavankumar-nm9yu]

Thank You Adam, Amazing Explanation...!!!

[@AdamMarczakYT]

My pleasure!

[@checomax1980]

Thank you Adam, Excellent tutorial.

[@AdamMarczakYT]

Glad you enjoyed it!

[@vishwanathr3507]

Hi Adam Thank you for excellent Tutorial and your time. Good luck Cheers! Vishwanath

[@AdamMarczakYT]

Glad you liked it! :)

[@Modern_revolution]

Thank u so much to you that you have share this valuable information !

[@kalangarahouse]

nice tutorial...please continue making such videos

[@jayong2370]

Awesome tutorial! Thank you.

[@BojackCartman]

nice tutorial, very clear and concise

[@AdamMarczakYT]

Glad you think so!

[@MrVoltelen]

Thanks for the great explanation

[@AdamMarczakYT]

My pleasure!

[@joseantonioestebanrodrigue448]

Perfect explanation!

[@devopsexpert9906]

Excellent Video, Best tutoring.......

[@AdamMarczakYT]

Many many thanks

[@angelamessina7187]

Thanks a lot Adam this will be realy helpful for my today's task. Be blessed

[@AdamMarczakYT]

Great! Thanks Angela, stay safe!

[@malayku8781]

Very nicely explained Adam!

[@AdamMarczakYT]

Thank you kindly! :)

[@soumenchandra7633]

What a fascinating presentation !!!

[@AdamMarczakYT]

Thank you! :)

[@oscararmandocisnerosruvalc8503]

Your videos are amazing bro !!!!!

[@sagarnegi]

Great explanation. I understood everything.

[@AdamMarczakYT]

Excellent

[@docondelta]

This is really amazing tutorial .. Thank you

[@AdamMarczakYT]

Cheers! Always happy to help!

[@priyankapatel9461]

Really very helpful!

[@AdamMarczakYT]

Thanks!

[@TheSQLPro]

Great tutorial!

[@AdamMarczakYT]

Thank you!

[@MarkDiamond]

Excellent content, thank you so much for the effort!

[@asrarhaq3234]

Wonderful - practical and useful- Thank you

[@AdamMarczakYT]

Thanks! Glad you like it.

[@danceandmanymorewithprisha5599]

great explanation !

[@AdamMarczakYT]

Thank you!

[@fsfehico]

Very well structured content. Thank you

[@AdamMarczakYT]

Thank you, glad you enjoy it :)

[@Unbox747]

I love your videos!

[@GhernieM]

Thanks Adam! Very helpful

[@AdamMarczakYT]

My pleasure!

[@RohkeaNox]

Very helpful, thank you :) !

[@shaileshsondawale2811]

Great work...!!!!

[@jayantKadam100]

This is really wonderful tutorial and very useful thanks

[@AdamMarczakYT]

Thank you! :)

[@aks541]

Nice video - just a thought: if you'd have used a FOREACH loop and inside that your copy activity was there , you probably don't need to give the name as Cars etc. Because loop took both the files from lake iteratively and pass to the copy activity one by one.

[@AdamMarczakYT]

Thanks. Correct, I show this approach in my bulk copy video kzbin.info/www/bejne/gaSyY3l-hsl8gtU

[@lorenzodelsoldato6899]

Thank you, really good content

[@TH-fe1vs]

nice tutorial Adam, thanks. However, I have some questions on the parameterization with Parameters and Variables. From the video, I see that there is still a need to manually key in those filesnames (cars, planes) in those steps...so if we have 50 of them, we need to that 50 times also? Is there a way to automatically detect those filenames and put those filenames in a loop?

[@javaguitarist]

I saw that Adam answered this question earlier: Yes you can! If 200 files have the same format then use prefix/wildcard path to pick up multiple files, if they are not then use Lookup activity then loop using ForEach activity. Thanks for watching.

[@santanughosal9785]

@@javaguitarist Do we have any video on this. I am having the same requirement but getting stuck in between

[@javaguitarist]

@@santanughosal9785 sorry, I do not know.

[@ricardorocha7882]

Simplesmente incrível esse vídeo. Vai me ajudar bastante.

[@oscarsalas6271]

Excelente video, me ayudo mucho

[@snehasiktachandra4357]

Thanks for the amazing videos ; Can you please give some idea how to design a pipeline for parallel processing of multiple SQL queries in ADF Pipeline ?

[@AdamMarczakYT]

It's too broad of a topic for a comment section. But try executing multiple by using execute stored procedure activity.

[@timeal2244]

Thank Adam, I appreciate your effort. You said "One cannot use User and password in production environment in Azure SQL Database", please can you show me how I can go about it in Production?

[@AdamMarczakYT]

For production using Managed Identity is advised. I'll prep the video on this in the future. For now simply check the docs, it's quite easy. Login to Azure SQL and run query CREATE USER 'adf_name' FROM EXTERNAL PROVIDER; EXEC sp_addrolemember 'db_datareader', 'adf_name'; and switch to data factory portal and in the linked service just change to managed identity instead of SQL authentication

[@Bremerhaven_Rocks]

What a great video. One question though. Does the Copy Data Activity automatically map the columns of the csv to the columns of the sql table automatically based on the column names? What is if the column names are different between the csv and sql tables?

[@AdamMarczakYT]

Thanks! If they are different it will throw error, but as a workaround you can create SQL view with the right column names. SQL server allows to perform inserts on the view :)

[@amitsingh8412]

Thanks Adam.

[@AdamMarczakYT]

My pleasure!

[@juliank5227]

Is it possible to parameterize a generic SAP Table Data Set and passing new values from an config file via an LookUp and ForEach Activity into a nested copy acitvity. We trying to avoid to create hundreds of Data Sets for each SAP table. Would be amazing to get an additional hint! Keep up the great tutorials.

[@sumashree6351]

Thank you very much good explanation.i have one dout which parameter used to improve the performance of the company for the past 6 years

[@juliestudy1475]

This is great! Instead of typing in the source and destination one at a time, is there a way to simply loop through the files folder, pick up the filenames and pass them as input (filename) parameters and use them again as output (tablename) parameter?

[@amit260183]

Hi Adam, Quite interesting way of elaborating the Parameterized concept, Question though... what if have to load 10 files in SQL table and DON't want to create action copy for every table, also don't want to execute 10 times after changing parameter value, it should be like packages will execute and load data for every table one by one based on Parameter value, i.e. iterative load based on named of file to SQL table. Thanks Amit

[@AdamMarczakYT]

Hey Amit. I have similar example posted as a video tutorial already, feel free to check it out kzbin.info/www/bejne/gaSyY3l-hsl8gtU it would just need a change of direction.

[@YanadireddyBonamukkala]

If possible try a video on crating Global parameters and pass the values dynamically with different DataBases

[@AdamMarczakYT]

I might do that in future, thanks.!

[@qweasdzxc2007]

@@AdamMarczakYT Hello! Is there a video for GP available now on your channel? ;)

[@bookko2377]

Thank you From Thailand

[@AdamMarczakYT]

My pleasure! :)

[@jonahnio5642]

Thanks alot Adam for the video. WOuld you be able to show me how I can pass variables across different pipelines? Thanks very much

[@AdamMarczakYT]

Thanks. Check out MS blog entry on that cloudblogs.microsoft.com/industry-blog/en-gb/technetuk/2020/03/19/enterprise-wide-orchestration-using-multiple-data-factories/

[@vickygana9260]

Thanks Adam for the vedio, it would have been great if you would have also showed how to pass the filename, tablename for multiple items(cars, planes) dynamically instead of passing manually while debugging.

[@AdamMarczakYT]

I did in my dynamic bulk load and triggers video, feel free to check then out kzbin.info/www/bejne/gaSyY3l-hsl8gtU and kzbin.info/www/bejne/q3eWfYJ5i6t7odk

[@_indrid_cold_]

Best content! Thank you so much!

[@AdamMarczakYT]

Awesome, cheers!

[@ambrishl9273]

Thanks a lot, This was very helpful!!!

[@AdamMarczakYT]

Thank you :)

[@Peter-cd9rp]

cool dude. Especially pipeline parameters! :)I can easily start pipelines with logic apps! :D Wohoo

[@AdamMarczakYT]

Glad you liked it!

[@sougatasil6064]

Thanks for the demo. But what to do in case of scheduled execution , because it is not possible to change the parameters manually during scheduled execution. Is there a way to pass the parameter dynamically so that , once it is scheduled for car and next time scheduled for plane without manual intervention

[@H2B-World]

Hi Adam, it looks flawless but I have a doubt in the present scenario. Without passing names manually in parameters while can't it read the file one by one and process accordingly?

[@louisar4227]

This was great, but is there a way to pass the original file name through to the source SQL table? My team receives thousands of excel files from suppliers. Is there a way to pass that original file name to the sink SQL table??

[@atulsalunke6675]

Awesome content and delivery Adam !! Thanks!! Please if possible add real Project based content using ADF

[@AdamMarczakYT]

Great suggestion! I do plan to have few videos on actual implementations :)

[@rakeshdey6970]

Awesome video, It will be great if you give demo on after building this pipeline with parameter how to deploy same thing in prod, Any json actually getting generated after building the pipeline?

[@AdamMarczakYT]

Thank you :) I do want to show CI/CD and deployments for ADF in future ;) but for now just export the template using ADF portal, change variables and deploy to prod.

[@Frederickawuahgyasi]

Great tutorial Adam. You have a student here. In trying to do this demo. I have realized that now the Linked Service has to be created whiles creating the Dataset. Is this a new functionality in ADF. Connections are not a available to create linked service separately .

[@AdamMarczakYT]

It was always like that you can create datsets from pipeline level, linked services from dataset level etc. But I found this to be very confusing for people. Only difference is now that Microsoft moved linked services to management tab on the left hand side. Thanks for watching ;)

[@Frederickawuahgyasi]

@@AdamMarczakYT Thanks Adam. Will check it out.

[@saikatsengupta6675]

Adam, the tutorial is great, concise and to the point. I have couple of questions regarding a full blown data pipeline. Is it possible to get in touch over email?

[@AdamMarczakYT]

Thanks for watching. Unfortunately I don't help offline as I don't have capacity to do so. You are always free to ask but if it's complex question I won't be able to help out. Thanks again for stopping by!

[@saikatsengupta6675]

@@AdamMarczakYT Thanks for the clarification. Wanted to check on below points: 1. How the parameterize strategy can help in a full blown data mart migration from on-premise to cloud as mentioned over the video? You definitely can't have a single for-each pipeline that can migrate the full data for a particular database on cloud in a real scenario. Some strategy tips will be really helpful. 2. What can be a possible solution to migrate large fact tables with approx 5 TB data? Do we need to have some custom parameter table that hold partition ranges which we can call in a lookup activity and do a for-each load? Any suggestion is really appreciated.

[@luisabaiano8689]

Thank you! From Italy

[@AdamMarczakYT]

And thank You for watching Luisa :)

[@tarvinder91]

very informative and helpful video

[@AdamMarczakYT]

Glad it was helpful!

[@kavishv.8101]

Do we have to always create individual table in SQL database for every csv file we copy from blob to database? Also, you've an amazing content, Thank you!

[@ivanovdenys]

Thanks a lot !!! Really really good !!

[@AdamMarczakYT]

Thanks! And thank you for watching Denys :)

[@yyy168-wt1zq]

Hi Adam, thanks for the video, that's a clear explanation, can I ask what if I want to put Cars and Planes into a variable array, then Pipeline can have a for loop execution of doing the copy of Cars and then Planes. Is it possible?

[@srinivasareddy3336]

Hi Adam, thanks for the great video tutorial. May I know how to pass variable values from one pipeline to another pipeline in Azure Data Factory.

[@mumair1979]

Hi Adam, This is fantastic. How can we take parameters from a SQL table, by using a stored procedure, instead of putting them in each time into the Datasources ?? can you please advise.. Thanks

[@rajexpert8544]

Hi Adam, Nice tutorial. How can I parameterize the pipeline if I want to loop through all the containers in blob storage and with the corresponding sql table name? I don't want to enter parameter values manually at run time

[@maxgobel]

HI Adam, thanks for the great tutorial, was very helpful. Would I be able to follow this procedure with a dataflow instead of a copy example? I am struggling with implementing this between a rawinputBlob and a cleanedOutputBlob

[@AdamMarczakYT]

It's a little different but mapping data flows support parametrization too: docs.microsoft.com/en-us/azure/data-factory/parameters-data-flow?WT.mc_id=AZ-MVP-5003556 thanks for stopping by!

[@dmsanz_youtube]

Great tutorial! Is there any option to export all the data factory and activity configuration so that we can create pipelines through ADF api programatically?

[@dmsanz_youtube]

I've just found out the export ARM template option includes everything in a json format

[@AdamMarczakYT]

Great! :)

[@naturelovervr5231]

Thanks a lot Adam for these videos .....can you please provide us diagram to explain data factory as it's not completely visible in KZbin videos please

[@AdamMarczakYT]

Thanks for stopping by. Not sure what is 'not visible' please elaborate.

[@farshadmardani7401]

Hi, if each destination table has different columns to those of source, then is it still possible to parameterize source and destination names? At present, I have to work in mappings tab for each of the sink tables which I guess because of different mappings then parameterization can not be done? Thankyou

[@AdamMarczakYT]

Yes. Dynamic mapping are pretty tricky but it's possible. One workaround that I like to do it to make it easier is to create view in SQL database which has the same format as that file and then set it as target. Because SQL allows to run inserts/updates over view it would work just fine.

[@amitjaju3351]

Hello Sir , I saw all your videos related to Azure Data Factory and all are very good . I am working on Azure to do some automation and I need your some help so can you please guide me on how can i handle scenario. I have a requirement like whenever the particular existing blob file get modify then the SP from SQL DB should get trigger and i have created ADF job to execute the stored procedure in azure sql db. In event based trigger we have only 2 options created/deleted of blob file. Can we have any alternate way to active this to do the automation. Thanks in advance and waiting for your reply.

[@AdamMarczakYT]

Hey Amit! Thanks :). Blob created event also triggers on file modification so you are good to go. Just add execute store procedure action and that's it! Documentation says "Triggered when a blob is created or replaced." More info on the event here docs.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage

[@tanmoybhattacharjee4966]

thank you, amazing tutorials , really helpful

[@AdamMarczakYT]

Thanks! :)