Are you real? You really explain and show demo in such a flawless manner. Thanks you so much, your Azure videos are happy at every step.
@AdamMarczakYT4 жыл бұрын
Thank you so much :)
@nicholasgregory47743 жыл бұрын
Another great one Adam, consider me a new student! Short enough in length while fitting all the relevant information in + easy to digest, without overly complicating things. Your PowerPoints with your explanations are second to none.
@AdamMarczakYT3 жыл бұрын
Awesome, thank you!
@eerosiljander4622 Жыл бұрын
Thanks Adam for these hands on videos. Best wishes,ES, Data engineer & Scientist.
@carl33p4 жыл бұрын
Demos are the best way to learn, and your demo is as good as it gets. Build, run, build, run- the incremental style allows viewer to understand slowly.
@AdamMarczakYT4 жыл бұрын
Couldn't agree more! Thanks!
@nikhilpatil20214 жыл бұрын
Thank you Adam, this is the best tutorial on Azure Data Factory I have seen! I am definitely going to try these demos. Thank you!
@AdamMarczakYT4 жыл бұрын
Great to hear! Cheers!
@rajdeepmandal14704 жыл бұрын
The quarantine days wouldn't be this useful without your teaching :) thank you Adam. You are the best!!!
@AdamMarczakYT4 жыл бұрын
Wow, thank you! :)
@nupoorsingh33233 жыл бұрын
It's great support for people like me who do not have a mentor. Thanks for uploading these videos and helping me to understand the workflow and survive.
@ghay33 жыл бұрын
My goto KZbin channel for anything Azure - you are doing the greatest service by sharing your knowledge and responding to all the queries! many thanks, Adam. Hope you can also enlighten the community on AWS :)
@AdamMarczakYT3 жыл бұрын
Wow, thank you! glad to help!
@tatianebrandao54879 ай бұрын
I'm just starting with Synapse/DF and I found your video extremely helpful for beginners. I managed to go through your video, complete the exercise on my own and everything made so much sense! Your way of explaining is very clear! Thank you very much for creating the video :)
@sankaji204 жыл бұрын
You are doing great job ADAM, 20-30 minutes videos of your's is equal to 60minuts of others. And quality of your vdo is un match-able. Now I am GCP guy but also very much interested in learning azure from your videos only.. keep uploading azure videos. Thanks so much.. doing wonderful job for learners.. you will get a lot of blessings.. Thumbs-up for you..
@AdamMarczakYT4 жыл бұрын
Wow, thanks! Appreciated!
@sraoarjun3 жыл бұрын
One word - FANTASTIC !! ... Thank you Adam , keep up the good work in educating several people .Also request you to put some videos on Synapse analytics ...
@AdamMarczakYT3 жыл бұрын
Thanks! Will do!
@mayur68163 жыл бұрын
Thanks, Adam. This is the best tutorial on ADF.
@AdamMarczakYT3 жыл бұрын
Glad you think so! Thanks!
@mashagalitskaia86424 ай бұрын
Thanks a ton for this video! It helped me a lot. Very precise and detailed, I had no questions left:)
@dabay2004 жыл бұрын
This is a really amazing video, so much better than Microsoft's tutorials.
@AdamMarczakYT4 жыл бұрын
Thanks again Dinal. You are too kind :).
@johnd36013 жыл бұрын
Thanks this is a big help, its amazing how much you don't know you know. This has opened up a lot of possible automation and saved me loads of time once I manage to apply this to my situation.
@TeymurRzayev3 жыл бұрын
Actually liked it before watching the video ;) Brilliant content, perfect explanation!
@barryjohnson89154 жыл бұрын
Really excellent tutorials. Very simply explained, easy to follow, fantastic stuff.
@AdamMarczakYT4 жыл бұрын
Thanks Barry :) Glad to have you here.
@eatingnetwork64743 жыл бұрын
great video, once again very clear and concise, thanks Adam
@AdamMarczakYT3 жыл бұрын
Thanks 👍
@arnabmondal55964 жыл бұрын
Awesome clip. Currently, better content than many available now.
@AdamMarczakYT4 жыл бұрын
Thank you so much :) glad that you enjoyed it.
@srinivasreddysankasani51403 жыл бұрын
Thank you Adam for your videos. your explanation is really awesome. looking for more videos
@AdamMarczakYT3 жыл бұрын
Glad you like them!
@jagadishparale86324 жыл бұрын
Hey Adam,It was excellent tutorial.I would like you to do more videos on deployment of Azure Data Factory pipelines when we use parameters
@AdamMarczakYT4 жыл бұрын
Thanks! You are in luck, new video tomorrow is also on ADF, stay tuned!
@snmailist14703 жыл бұрын
Thank you Adam. I'm very appreciate your effort 👍👍👍👍 You give me a description what's going on the cloud.
@AdamMarczakYT3 жыл бұрын
Glad to hear it , thank you!
@sazidaanjumsheikh68172 жыл бұрын
Thank you Adam, in a very short video you made me understand the ADF with a demo instead of Theory.. Nice content... :-)
@iChrisBirch8 ай бұрын
Great tutorial, answered my questions as I had them, great logical flow.
@santanughosal97852 жыл бұрын
Awesome, simple and hands on video to understand the concept. Thank you
@lehai91723 жыл бұрын
Fantastic tutorials, I've learned a lot from your videos. Thank you for sharing your experiences, Adam.
@AdamMarczakYT3 жыл бұрын
My pleasure!
@baladenmark3 жыл бұрын
Wow. Thank you Adam. I really can't thank you enough.
@AdamMarczakYT3 жыл бұрын
Glad I could help!
@ashishjuneja65742 жыл бұрын
Super awesome !! Great going Adam.
@igiq5 жыл бұрын
Keep it up Adam, your videos are very helpful !
@AdamMarczakYT5 жыл бұрын
Thanks! will do!
@joshiabhinav4 жыл бұрын
Man you are awesome. you present so well and explain so well. Love the way you teach !
@AdamMarczakYT4 жыл бұрын
I appreciate that!
@SuperSanjays2 жыл бұрын
Great information you have shared Adam. It helps me a lot.
@venkataramakrishnamagapu76453 жыл бұрын
Excellent content. Simple and informative.
@AdamMarczakYT3 жыл бұрын
Much appreciated!
@IntelliCloudTech2 жыл бұрын
Clearly explained at every point. thanks for the video.
@pavankumar-nm9yu3 жыл бұрын
Thank You Adam, Amazing Explanation...!!!
@AdamMarczakYT3 жыл бұрын
My pleasure!
@checomax19803 жыл бұрын
Thank you Adam, Excellent tutorial.
@AdamMarczakYT3 жыл бұрын
Glad you enjoyed it!
@vishwanathr35074 жыл бұрын
Hi Adam Thank you for excellent Tutorial and your time. Good luck Cheers! Vishwanath
@AdamMarczakYT4 жыл бұрын
Glad you liked it! :)
@Modern_revolution2 жыл бұрын
Thank u so much to you that you have share this valuable information !
@kalangarahouse2 жыл бұрын
nice tutorial...please continue making such videos
@jayong23702 жыл бұрын
Awesome tutorial! Thank you.
@BojackCartman3 жыл бұрын
nice tutorial, very clear and concise
@AdamMarczakYT3 жыл бұрын
Glad you think so!
@MrVoltelen3 жыл бұрын
Thanks for the great explanation
@AdamMarczakYT3 жыл бұрын
My pleasure!
@joseantonioestebanrodrigue4483 жыл бұрын
Perfect explanation!
@devopsexpert99063 жыл бұрын
Excellent Video, Best tutoring.......
@AdamMarczakYT3 жыл бұрын
Many many thanks
@angelamessina71874 жыл бұрын
Thanks a lot Adam this will be realy helpful for my today's task. Be blessed
@AdamMarczakYT4 жыл бұрын
Great! Thanks Angela, stay safe!
@malayku87814 жыл бұрын
Very nicely explained Adam!
@AdamMarczakYT4 жыл бұрын
Thank you kindly! :)
@soumenchandra76334 жыл бұрын
What a fascinating presentation !!!
@AdamMarczakYT4 жыл бұрын
Thank you! :)
@oscararmandocisnerosruvalc8503 Жыл бұрын
Your videos are amazing bro !!!!!
@sagarnegi3 жыл бұрын
Great explanation. I understood everything.
@AdamMarczakYT3 жыл бұрын
Excellent
@docondelta4 жыл бұрын
This is really amazing tutorial .. Thank you
@AdamMarczakYT4 жыл бұрын
Cheers! Always happy to help!
@priyankapatel94613 жыл бұрын
Really very helpful!
@AdamMarczakYT3 жыл бұрын
Thanks!
@TheSQLPro3 жыл бұрын
Great tutorial!
@AdamMarczakYT3 жыл бұрын
Thank you!
@MarkDiamond3 жыл бұрын
Excellent content, thank you so much for the effort!
@asrarhaq32344 жыл бұрын
Wonderful - practical and useful- Thank you
@AdamMarczakYT4 жыл бұрын
Thanks! Glad you like it.
@danceandmanymorewithprisha55994 жыл бұрын
great explanation !
@AdamMarczakYT4 жыл бұрын
Thank you!
@fsfehico4 жыл бұрын
Very well structured content. Thank you
@AdamMarczakYT4 жыл бұрын
Thank you, glad you enjoy it :)
@Unbox7472 жыл бұрын
I love your videos!
@GhernieM4 жыл бұрын
Thanks Adam! Very helpful
@AdamMarczakYT4 жыл бұрын
My pleasure!
@RohkeaNox Жыл бұрын
Very helpful, thank you :) !
@shaileshsondawale2811 Жыл бұрын
Great work...!!!!
@jayantKadam1004 жыл бұрын
This is really wonderful tutorial and very useful thanks
@AdamMarczakYT4 жыл бұрын
Thank you! :)
@aks5413 жыл бұрын
Nice video - just a thought: if you'd have used a FOREACH loop and inside that your copy activity was there , you probably don't need to give the name as Cars etc. Because loop took both the files from lake iteratively and pass to the copy activity one by one.
@AdamMarczakYT3 жыл бұрын
Thanks. Correct, I show this approach in my bulk copy video kzbin.info/www/bejne/gaSyY3l-hsl8gtU
@lorenzodelsoldato68996 ай бұрын
Thank you, really good content
@TH-fe1vs2 жыл бұрын
nice tutorial Adam, thanks. However, I have some questions on the parameterization with Parameters and Variables. From the video, I see that there is still a need to manually key in those filesnames (cars, planes) in those steps...so if we have 50 of them, we need to that 50 times also? Is there a way to automatically detect those filenames and put those filenames in a loop?
@javaguitarist2 жыл бұрын
I saw that Adam answered this question earlier: Yes you can! If 200 files have the same format then use prefix/wildcard path to pick up multiple files, if they are not then use Lookup activity then loop using ForEach activity. Thanks for watching.
@santanughosal97852 жыл бұрын
@@javaguitarist Do we have any video on this. I am having the same requirement but getting stuck in between
@javaguitarist2 жыл бұрын
@@santanughosal9785 sorry, I do not know.
@ricardorocha7882 Жыл бұрын
Simplesmente incrível esse vídeo. Vai me ajudar bastante.
@oscarsalas62713 жыл бұрын
Excelente video, me ayudo mucho
@snehasiktachandra43573 жыл бұрын
Thanks for the amazing videos ; Can you please give some idea how to design a pipeline for parallel processing of multiple SQL queries in ADF Pipeline ?
@AdamMarczakYT3 жыл бұрын
It's too broad of a topic for a comment section. But try executing multiple by using execute stored procedure activity.
@timeal22443 жыл бұрын
Thank Adam, I appreciate your effort. You said "One cannot use User and password in production environment in Azure SQL Database", please can you show me how I can go about it in Production?
@AdamMarczakYT3 жыл бұрын
For production using Managed Identity is advised. I'll prep the video on this in the future. For now simply check the docs, it's quite easy. Login to Azure SQL and run query CREATE USER 'adf_name' FROM EXTERNAL PROVIDER; EXEC sp_addrolemember 'db_datareader', 'adf_name'; and switch to data factory portal and in the linked service just change to managed identity instead of SQL authentication
@Bremerhaven_Rocks3 жыл бұрын
What a great video. One question though. Does the Copy Data Activity automatically map the columns of the csv to the columns of the sql table automatically based on the column names? What is if the column names are different between the csv and sql tables?
@AdamMarczakYT3 жыл бұрын
Thanks! If they are different it will throw error, but as a workaround you can create SQL view with the right column names. SQL server allows to perform inserts on the view :)
@amitsingh84124 жыл бұрын
Thanks Adam.
@AdamMarczakYT4 жыл бұрын
My pleasure!
@juliank52272 жыл бұрын
Is it possible to parameterize a generic SAP Table Data Set and passing new values from an config file via an LookUp and ForEach Activity into a nested copy acitvity. We trying to avoid to create hundreds of Data Sets for each SAP table. Would be amazing to get an additional hint! Keep up the great tutorials.
@sumashree63512 жыл бұрын
Thank you very much good explanation.i have one dout which parameter used to improve the performance of the company for the past 6 years
@juliestudy14759 ай бұрын
This is great! Instead of typing in the source and destination one at a time, is there a way to simply loop through the files folder, pick up the filenames and pass them as input (filename) parameters and use them again as output (tablename) parameter?
@amit2601834 жыл бұрын
Hi Adam, Quite interesting way of elaborating the Parameterized concept, Question though... what if have to load 10 files in SQL table and DON't want to create action copy for every table, also don't want to execute 10 times after changing parameter value, it should be like packages will execute and load data for every table one by one based on Parameter value, i.e. iterative load based on named of file to SQL table. Thanks Amit
@AdamMarczakYT4 жыл бұрын
Hey Amit. I have similar example posted as a video tutorial already, feel free to check it out kzbin.info/www/bejne/gaSyY3l-hsl8gtU it would just need a change of direction.
@YanadireddyBonamukkala3 жыл бұрын
If possible try a video on crating Global parameters and pass the values dynamically with different DataBases
@AdamMarczakYT3 жыл бұрын
I might do that in future, thanks.!
@qweasdzxc20073 жыл бұрын
@@AdamMarczakYT Hello! Is there a video for GP available now on your channel? ;)
@bookko23774 жыл бұрын
Thank you From Thailand
@AdamMarczakYT4 жыл бұрын
My pleasure! :)
@jonahnio56424 жыл бұрын
Thanks alot Adam for the video. WOuld you be able to show me how I can pass variables across different pipelines? Thanks very much
@AdamMarczakYT4 жыл бұрын
Thanks. Check out MS blog entry on that cloudblogs.microsoft.com/industry-blog/en-gb/technetuk/2020/03/19/enterprise-wide-orchestration-using-multiple-data-factories/
@vickygana92604 жыл бұрын
Thanks Adam for the vedio, it would have been great if you would have also showed how to pass the filename, tablename for multiple items(cars, planes) dynamically instead of passing manually while debugging.
@AdamMarczakYT4 жыл бұрын
I did in my dynamic bulk load and triggers video, feel free to check then out kzbin.info/www/bejne/gaSyY3l-hsl8gtU and kzbin.info/www/bejne/q3eWfYJ5i6t7odk
@_indrid_cold_4 жыл бұрын
Best content! Thank you so much!
@AdamMarczakYT4 жыл бұрын
Awesome, cheers!
@ambrishl92734 жыл бұрын
Thanks a lot, This was very helpful!!!
@AdamMarczakYT4 жыл бұрын
Thank you :)
@Peter-cd9rp4 жыл бұрын
cool dude. Especially pipeline parameters! :)I can easily start pipelines with logic apps! :D Wohoo
@AdamMarczakYT4 жыл бұрын
Glad you liked it!
@sougatasil60642 жыл бұрын
Thanks for the demo. But what to do in case of scheduled execution , because it is not possible to change the parameters manually during scheduled execution. Is there a way to pass the parameter dynamically so that , once it is scheduled for car and next time scheduled for plane without manual intervention
@H2B-World2 жыл бұрын
Hi Adam, it looks flawless but I have a doubt in the present scenario. Without passing names manually in parameters while can't it read the file one by one and process accordingly?
@louisar4227Ай бұрын
This was great, but is there a way to pass the original file name through to the source SQL table? My team receives thousands of excel files from suppliers. Is there a way to pass that original file name to the sink SQL table??
@atulsalunke66753 жыл бұрын
Awesome content and delivery Adam !! Thanks!! Please if possible add real Project based content using ADF
@AdamMarczakYT3 жыл бұрын
Great suggestion! I do plan to have few videos on actual implementations :)
@rakeshdey69703 жыл бұрын
Awesome video, It will be great if you give demo on after building this pipeline with parameter how to deploy same thing in prod, Any json actually getting generated after building the pipeline?
@AdamMarczakYT3 жыл бұрын
Thank you :) I do want to show CI/CD and deployments for ADF in future ;) but for now just export the template using ADF portal, change variables and deploy to prod.
@Frederickawuahgyasi4 жыл бұрын
Great tutorial Adam. You have a student here. In trying to do this demo. I have realized that now the Linked Service has to be created whiles creating the Dataset. Is this a new functionality in ADF. Connections are not a available to create linked service separately .
@AdamMarczakYT4 жыл бұрын
It was always like that you can create datsets from pipeline level, linked services from dataset level etc. But I found this to be very confusing for people. Only difference is now that Microsoft moved linked services to management tab on the left hand side. Thanks for watching ;)
@Frederickawuahgyasi4 жыл бұрын
@@AdamMarczakYT Thanks Adam. Will check it out.
@saikatsengupta66754 жыл бұрын
Adam, the tutorial is great, concise and to the point. I have couple of questions regarding a full blown data pipeline. Is it possible to get in touch over email?
@AdamMarczakYT4 жыл бұрын
Thanks for watching. Unfortunately I don't help offline as I don't have capacity to do so. You are always free to ask but if it's complex question I won't be able to help out. Thanks again for stopping by!
@saikatsengupta66754 жыл бұрын
@@AdamMarczakYT Thanks for the clarification. Wanted to check on below points: 1. How the parameterize strategy can help in a full blown data mart migration from on-premise to cloud as mentioned over the video? You definitely can't have a single for-each pipeline that can migrate the full data for a particular database on cloud in a real scenario. Some strategy tips will be really helpful. 2. What can be a possible solution to migrate large fact tables with approx 5 TB data? Do we need to have some custom parameter table that hold partition ranges which we can call in a lookup activity and do a for-each load? Any suggestion is really appreciated.
@luisabaiano86894 жыл бұрын
Thank you! From Italy
@AdamMarczakYT4 жыл бұрын
And thank You for watching Luisa :)
@tarvinder914 жыл бұрын
very informative and helpful video
@AdamMarczakYT4 жыл бұрын
Glad it was helpful!
@kavishv.81012 жыл бұрын
Do we have to always create individual table in SQL database for every csv file we copy from blob to database? Also, you've an amazing content, Thank you!
@ivanovdenys4 жыл бұрын
Thanks a lot !!! Really really good !!
@AdamMarczakYT4 жыл бұрын
Thanks! And thank you for watching Denys :)
@yyy168-wt1zq5 ай бұрын
Hi Adam, thanks for the video, that's a clear explanation, can I ask what if I want to put Cars and Planes into a variable array, then Pipeline can have a for loop execution of doing the copy of Cars and then Planes. Is it possible?
@srinivasareddy33363 жыл бұрын
Hi Adam, thanks for the great video tutorial. May I know how to pass variable values from one pipeline to another pipeline in Azure Data Factory.
@mumair19793 жыл бұрын
Hi Adam, This is fantastic. How can we take parameters from a SQL table, by using a stored procedure, instead of putting them in each time into the Datasources ?? can you please advise.. Thanks
@rajexpert85442 жыл бұрын
Hi Adam, Nice tutorial. How can I parameterize the pipeline if I want to loop through all the containers in blob storage and with the corresponding sql table name? I don't want to enter parameter values manually at run time
@maxgobel3 жыл бұрын
HI Adam, thanks for the great tutorial, was very helpful. Would I be able to follow this procedure with a dataflow instead of a copy example? I am struggling with implementing this between a rawinputBlob and a cleanedOutputBlob
@AdamMarczakYT3 жыл бұрын
It's a little different but mapping data flows support parametrization too: docs.microsoft.com/en-us/azure/data-factory/parameters-data-flow?WT.mc_id=AZ-MVP-5003556 thanks for stopping by!
@dmsanz_youtube4 жыл бұрын
Great tutorial! Is there any option to export all the data factory and activity configuration so that we can create pipelines through ADF api programatically?
@dmsanz_youtube4 жыл бұрын
I've just found out the export ARM template option includes everything in a json format
@AdamMarczakYT4 жыл бұрын
Great! :)
@naturelovervr52313 жыл бұрын
Thanks a lot Adam for these videos .....can you please provide us diagram to explain data factory as it's not completely visible in KZbin videos please
@AdamMarczakYT3 жыл бұрын
Thanks for stopping by. Not sure what is 'not visible' please elaborate.
@farshadmardani74014 жыл бұрын
Hi, if each destination table has different columns to those of source, then is it still possible to parameterize source and destination names? At present, I have to work in mappings tab for each of the sink tables which I guess because of different mappings then parameterization can not be done? Thankyou
@AdamMarczakYT4 жыл бұрын
Yes. Dynamic mapping are pretty tricky but it's possible. One workaround that I like to do it to make it easier is to create view in SQL database which has the same format as that file and then set it as target. Because SQL allows to run inserts/updates over view it would work just fine.
@amitjaju33514 жыл бұрын
Hello Sir , I saw all your videos related to Azure Data Factory and all are very good . I am working on Azure to do some automation and I need your some help so can you please guide me on how can i handle scenario. I have a requirement like whenever the particular existing blob file get modify then the SP from SQL DB should get trigger and i have created ADF job to execute the stored procedure in azure sql db. In event based trigger we have only 2 options created/deleted of blob file. Can we have any alternate way to active this to do the automation. Thanks in advance and waiting for your reply.
@AdamMarczakYT4 жыл бұрын
Hey Amit! Thanks :). Blob created event also triggers on file modification so you are good to go. Just add execute store procedure action and that's it! Documentation says "Triggered when a blob is created or replaced." More info on the event here docs.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage