Azure Data Factory Parametrization Tutorial

[@raviv5109]

Are you real? You really explain and show demo in such a flawless manner. Thanks you so much, your Azure videos are happy at every step.

[@AdamMarczakYT]

Thank you so much :)

[@nicholasgregory4774]

Another great one Adam, consider me a new student! Short enough in length while fitting all the relevant information in + easy to digest, without overly complicating things. Your PowerPoints with your explanations are second to none.

[@AdamMarczakYT]

Awesome, thank you!

[@eerosiljander4622]

Thanks Adam for these hands on videos. Best wishes,ES, Data engineer & Scientist.

[@carl33p]

Demos are the best way to learn, and your demo is as good as it gets. Build, run, build, run- the incremental style allows viewer to understand slowly.

[@AdamMarczakYT]

Couldn't agree more! Thanks!

[@atlanticoceanvoyagebird2630]

Parameterization of its expiations which contains alternative methods of implementations required a complete young brains not like 76yrs old man. You are great to explain everything without hiding your knowledges.

[@nikhilpatil2021]

Thank you Adam, this is the best tutorial on Azure Data Factory I have seen! I am definitely going to try these demos. Thank you!

[@AdamMarczakYT]

Great to hear! Cheers!

[@nupoorsingh3323]

It's great support for people like me who do not have a mentor. Thanks for uploading these videos and helping me to understand the workflow and survive.

[@ghay3]

My goto KZbin channel for anything Azure - you are doing the greatest service by sharing your knowledge and responding to all the queries! many thanks, Adam. Hope you can also enlighten the community on AWS :)

[@AdamMarczakYT]

Wow, thank you! glad to help!

[@sankaji20]

You are doing great job ADAM, 20-30 minutes videos of your's is equal to 60minuts of others. And quality of your vdo is un match-able. Now I am GCP guy but also very much interested in learning azure from your videos only.. keep uploading azure videos. Thanks so much.. doing wonderful job for learners.. you will get a lot of blessings.. Thumbs-up for you..

[@AdamMarczakYT]

Wow, thanks! Appreciated!

[@tatianebrandao5487]

I'm just starting with Synapse/DF and I found your video extremely helpful for beginners. I managed to go through your video, complete the exercise on my own and everything made so much sense! Your way of explaining is very clear! Thank you very much for creating the video :)

[@rajdeepmandal1470]

The quarantine days wouldn't be this useful without your teaching :) thank you Adam. You are the best!!!

[@AdamMarczakYT]

Wow, thank you! :)

[@sraoarjun]

One word - FANTASTIC !! ... Thank you Adam , keep up the good work in educating several people .Also request you to put some videos on Synapse analytics ...

[@AdamMarczakYT]

Thanks! Will do!

[@mayur6816]

Thanks, Adam. This is the best tutorial on ADF.

[@AdamMarczakYT]

Glad you think so! Thanks!

[@srinivasreddysankasani5140]

Thank you Adam for your videos. your explanation is really awesome. looking for more videos

[@AdamMarczakYT]

Glad you like them!

[@TeymurRzayev]

Actually liked it before watching the video ;) Brilliant content, perfect explanation!

[@snmailist1470]

Thank you Adam. I'm very appreciate your effort 👍👍👍👍 You give me a description what's going on the cloud.

[@AdamMarczakYT]

Glad to hear it , thank you!

[@barryjohnson8915]

Really excellent tutorials. Very simply explained, easy to follow, fantastic stuff.

[@AdamMarczakYT]

Thanks Barry :) Glad to have you here.

[@johnd3601]

Thanks this is a big help, its amazing how much you don't know you know. This has opened up a lot of possible automation and saved me loads of time once I manage to apply this to my situation.

[@eatingnetwork6474]

great video, once again very clear and concise, thanks Adam

[@AdamMarczakYT]

Thanks 👍

[@dabay200]

This is a really amazing video, so much better than Microsoft's tutorials.

[@AdamMarczakYT]

Thanks again Dinal. You are too kind :).

[@mashagalitskaia8642]

Thanks a ton for this video! It helped me a lot. Very precise and detailed, I had no questions left:)

[@arnabmondal5596]

Awesome clip. Currently, better content than many available now.

[@AdamMarczakYT]

Thank you so much :) glad that you enjoyed it.

[@jagadishparale8632]

Hey Adam,It was excellent tutorial.I would like you to do more videos on deployment of Azure Data Factory pipelines when we use parameters

[@AdamMarczakYT]

Thanks! You are in luck, new video tomorrow is also on ADF, stay tuned!

[@santanughosal9785]

Awesome, simple and hands on video to understand the concept. Thank you

[@lehai9172]

Fantastic tutorials, I've learned a lot from your videos. Thank you for sharing your experiences, Adam.

[@AdamMarczakYT]

My pleasure!

[@joshiabhinav]

Man you are awesome. you present so well and explain so well. Love the way you teach !

[@AdamMarczakYT]

I appreciate that!

[@baladenmark]

Wow. Thank you Adam. I really can't thank you enough.

[@AdamMarczakYT]

Glad I could help!

[@venkataramakrishnamagapu7645]

Excellent content. Simple and informative.

[@AdamMarczakYT]

Much appreciated!

[@SuperSanjays]

Great information you have shared Adam. It helps me a lot.

[@sazidaanjumsheikh6817]

Thank you Adam, in a very short video you made me understand the ADF with a demo instead of Theory.. Nice content... :-)

[@iChrisBirch]

Great tutorial, answered my questions as I had them, great logical flow.

[@pavankumar-nm9yu]

Thank You Adam, Amazing Explanation...!!!

[@AdamMarczakYT]

My pleasure!

[@igiq]

Keep it up Adam, your videos are very helpful !

[@AdamMarczakYT]

Thanks! will do!

[@IntelliCloudTech]

Clearly explained at every point. thanks for the video.

[@ashishjuneja6574]

Super awesome !! Great going Adam.

[@MrVoltelen]

Thanks for the great explanation

[@AdamMarczakYT]

My pleasure!

[@devopsexpert9906]

Excellent Video, Best tutoring.......

[@AdamMarczakYT]

Many many thanks

[@BojackCartman]

nice tutorial, very clear and concise

[@AdamMarczakYT]

Glad you think so!

[@vishwanathr3507]

Hi Adam Thank you for excellent Tutorial and your time. Good luck Cheers! Vishwanath

[@AdamMarczakYT]

Glad you liked it! :)

[@checomax1980]

Thank you Adam, Excellent tutorial.

[@AdamMarczakYT]

Glad you enjoyed it!

[@malayku8781]

Very nicely explained Adam!

[@AdamMarczakYT]

Thank you kindly! :)

[@kalangarahouse]

nice tutorial...please continue making such videos

[@angelamessina7187]

Thanks a lot Adam this will be realy helpful for my today's task. Be blessed

[@AdamMarczakYT]

Great! Thanks Angela, stay safe!

[@soumenchandra7633]

What a fascinating presentation !!!

[@AdamMarczakYT]

Thank you! :)

[@Modern_revolution]

Thank u so much to you that you have share this valuable information !

[@joseantonioestebanrodrigue448]

Perfect explanation!

[@sagarnegi]

Great explanation. I understood everything.

[@AdamMarczakYT]

Excellent

[@TheSQLPro]

Great tutorial!

[@AdamMarczakYT]

Thank you!

[@jayong2370]

Awesome tutorial! Thank you.

[@YanadireddyBonamukkala]

If possible try a video on crating Global parameters and pass the values dynamically with different DataBases

[@AdamMarczakYT]

I might do that in future, thanks.!

[@qweasdzxc2007]

@@AdamMarczakYT Hello! Is there a video for GP available now on your channel? ;)

[@docondelta]

This is really amazing tutorial .. Thank you

[@AdamMarczakYT]

Cheers! Always happy to help!

[@oscararmandocisnerosruvalc8503]

Your videos are amazing bro !!!!!

[@danceandmanymorewithprisha5599]

great explanation !

[@AdamMarczakYT]

Thank you!

[@priyankapatel9461]

Really very helpful!

[@AdamMarczakYT]

Thanks!

[@MarkDiamond]

Excellent content, thank you so much for the effort!

[@fsfehico]

Very well structured content. Thank you

[@AdamMarczakYT]

Thank you, glad you enjoy it :)

[@asrarhaq3234]

Wonderful - practical and useful- Thank you

[@AdamMarczakYT]

Thanks! Glad you like it.

[@GhernieM]

Thanks Adam! Very helpful

[@AdamMarczakYT]

My pleasure!

[@dataengsafari]

Great video !

[@aks541]

Nice video - just a thought: if you'd have used a FOREACH loop and inside that your copy activity was there , you probably don't need to give the name as Cars etc. Because loop took both the files from lake iteratively and pass to the copy activity one by one.

[@AdamMarczakYT]

Thanks. Correct, I show this approach in my bulk copy video kzbin.info/www/bejne/gaSyY3l-hsl8gtU

[@jayantKadam100]

This is really wonderful tutorial and very useful thanks

[@AdamMarczakYT]

Thank you! :)

[@Rrabbit_swim]

I love your videos!

[@sumashree6351]

Thank you very much good explanation.i have one dout which parameter used to improve the performance of the company for the past 6 years

[@aperxmim]

Thanks this video is 5 years old when I am watching they moved fields around so if the student has to adjust to the newer versio.

[@shaileshsondawale2811]

Great work...!!!!

[@amitsingh8412]

Thanks Adam.

[@AdamMarczakYT]

My pleasure!

[@lorenzodelsoldato6899]

Thank you, really good content

[@bookko2377]

Thank you From Thailand

[@AdamMarczakYT]

My pleasure! :)

[@vickygana9260]

Thanks Adam for the vedio, it would have been great if you would have also showed how to pass the filename, tablename for multiple items(cars, planes) dynamically instead of passing manually while debugging.

[@AdamMarczakYT]

I did in my dynamic bulk load and triggers video, feel free to check then out kzbin.info/www/bejne/gaSyY3l-hsl8gtU and kzbin.info/www/bejne/q3eWfYJ5i6t7odk

[@tarvinder91]

very informative and helpful video

[@AdamMarczakYT]

Glad it was helpful!

[@atulsalunke6675]

Awesome content and delivery Adam !! Thanks!! Please if possible add real Project based content using ADF

[@AdamMarczakYT]

Great suggestion! I do plan to have few videos on actual implementations :)

[@emmanuelagbesola4618]

Hello Adam, thanks for this tutorial, although i think the scenarios have changed in the current version of ADF.

[@juliestudy1475]

This is great! Instead of typing in the source and destination one at a time, is there a way to simply loop through the files folder, pick up the filenames and pass them as input (filename) parameters and use them again as output (tablename) parameter?

[@oscarsalas6271]

Excelente video, me ayudo mucho

[@ricardorocha7882]

Simplesmente incrível esse vídeo. Vai me ajudar bastante.

[@timeal2244]

Thank Adam, I appreciate your effort. You said "One cannot use User and password in production environment in Azure SQL Database", please can you show me how I can go about it in Production?

[@AdamMarczakYT]

For production using Managed Identity is advised. I'll prep the video on this in the future. For now simply check the docs, it's quite easy. Login to Azure SQL and run query CREATE USER 'adf_name' FROM EXTERNAL PROVIDER; EXEC sp_addrolemember 'db_datareader', 'adf_name'; and switch to data factory portal and in the linked service just change to managed identity instead of SQL authentication

[@_indrid_cold_]

Best content! Thank you so much!

[@AdamMarczakYT]

Awesome, cheers!

[@juliank5227]

Is it possible to parameterize a generic SAP Table Data Set and passing new values from an config file via an LookUp and ForEach Activity into a nested copy acitvity. We trying to avoid to create hundreds of Data Sets for each SAP table. Would be amazing to get an additional hint! Keep up the great tutorials.

[@snehasiktachandra4357]

Thanks for the amazing videos ; Can you please give some idea how to design a pipeline for parallel processing of multiple SQL queries in ADF Pipeline ?

[@AdamMarczakYT]

It's too broad of a topic for a comment section. But try executing multiple by using execute stored procedure activity.

[@Bremerhaven_Rocks]

What a great video. One question though. Does the Copy Data Activity automatically map the columns of the csv to the columns of the sql table automatically based on the column names? What is if the column names are different between the csv and sql tables?

[@AdamMarczakYT]

Thanks! If they are different it will throw error, but as a workaround you can create SQL view with the right column names. SQL server allows to perform inserts on the view :)

[@ambrishl9273]

Thanks a lot, This was very helpful!!!

[@AdamMarczakYT]

Thank you :)

[@ivanovdenys]

Thanks a lot !!! Really really good !!

[@AdamMarczakYT]

Thanks! And thank you for watching Denys :)

[@sougatasil6064]

Thanks for the demo. But what to do in case of scheduled execution , because it is not possible to change the parameters manually during scheduled execution. Is there a way to pass the parameter dynamically so that , once it is scheduled for car and next time scheduled for plane without manual intervention

[@sameerdongare1113]

Awesome video

[@AdamMarczakYT]

Thaaaanks :)

[@geethamca88]

Excellent

[@Peter-cd9rp]

cool dude. Especially pipeline parameters! :)I can easily start pipelines with logic apps! :D Wohoo

[@AdamMarczakYT]

Glad you liked it!

[@rakeshdey6970]

Awesome video, It will be great if you give demo on after building this pipeline with parameter how to deploy same thing in prod, Any json actually getting generated after building the pipeline?

[@AdamMarczakYT]

Thank you :) I do want to show CI/CD and deployments for ADF in future ;) but for now just export the template using ADF portal, change variables and deploy to prod.

[@luisabaiano8689]

Thank you! From Italy

[@AdamMarczakYT]

And thank You for watching Luisa :)

[@jonahnio5642]

Thanks alot Adam for the video. WOuld you be able to show me how I can pass variables across different pipelines? Thanks very much

[@AdamMarczakYT]

Thanks. Check out MS blog entry on that cloudblogs.microsoft.com/industry-blog/en-gb/technetuk/2020/03/19/enterprise-wide-orchestration-using-multiple-data-factories/

[@amit260183]

Hi Adam, Quite interesting way of elaborating the Parameterized concept, Question though... what if have to load 10 files in SQL table and DON't want to create action copy for every table, also don't want to execute 10 times after changing parameter value, it should be like packages will execute and load data for every table one by one based on Parameter value, i.e. iterative load based on named of file to SQL table. Thanks Amit

[@AdamMarczakYT]

Hey Amit. I have similar example posted as a video tutorial already, feel free to check it out kzbin.info/www/bejne/gaSyY3l-hsl8gtU it would just need a change of direction.

[@naturelovervr5231]

Thanks a lot Adam for these videos .....can you please provide us diagram to explain data factory as it's not completely visible in KZbin videos please

[@AdamMarczakYT]

Thanks for stopping by. Not sure what is 'not visible' please elaborate.

[@alfredsfutterkiste7534]

Love it

[@louisar4227]

This was great, but is there a way to pass the original file name through to the source SQL table? My team receives thousands of excel files from suppliers. Is there a way to pass that original file name to the sink SQL table??

[@TheDataArchitect]

What about using Objects as params? and how to get their values for each Copy Activity?

[@YanadireddyBonamukkala]

Is there any possible to change the linked service dynamically at triggering level, if yes please share your video here

[@AdamMarczakYT]

Yep, check this out docs.microsoft.com/en-us/azure/data-factory/parameterize-linked-services?WT.mc_id=AZ-MVP-5003556 but I don't have video on that yet.

[@Frederickawuahgyasi]

Great tutorial Adam. You have a student here. In trying to do this demo. I have realized that now the Linked Service has to be created whiles creating the Dataset. Is this a new functionality in ADF. Connections are not a available to create linked service separately .

[@AdamMarczakYT]

It was always like that you can create datsets from pipeline level, linked services from dataset level etc. But I found this to be very confusing for people. Only difference is now that Microsoft moved linked services to management tab on the left hand side. Thanks for watching ;)

[@Frederickawuahgyasi]

@@AdamMarczakYT Thanks Adam. Will check it out.

[@yyy168-wt1zq]

Hi Adam, thanks for the video, that's a clear explanation, can I ask what if I want to put Cars and Planes into a variable array, then Pipeline can have a for loop execution of doing the copy of Cars and then Planes. Is it possible?

[@leonkriner3744]

very helpful. Thank you!

[@AdamMarczakYT]

Glad you liked it!

[@tanmoybhattacharjee4966]

thank you, amazing tutorials , really helpful

[@AdamMarczakYT]

Thanks! :)

[@rajexpert8544]

Hi Adam, Nice tutorial. How can I parameterize the pipeline if I want to loop through all the containers in blob storage and with the corresponding sql table name? I don't want to enter parameter values manually at run time

[@shruthil7913]

Hi Adam, There is any way to move the data from cars and plane data set to SQL tables, instead of providing individually cars and planes while debug.

[@AdamMarczakYT]

Yep! You can use For-each loops and create flows which allow you to import multiple. GetMetadata for blob to SQL and Lookup for SQL to blob flows. :)

[@shruthil7913]

@@AdamMarczakYT Thank you... Please one video..