76. Continuous integration and deployment in Azure Data Factory

[@empowerpeoplebytech2347]

Awesome, this is very important from CI/CD perspective, must know for every ADF developer, without this you cannot give end to end solution. Thank you very much for this video. I struggled a lot, but put it together, tried and finally it worked, very happy. Thanks once again!!

[@WafaStudies]

Thank you 🙂

[@empowerpeoplebytech2347]

I had been looking for a comprehensive guide for ADF, but was unable to find one, finally i landed on your channel and found this vast content on ADF. Many thanks, whenever i forget something i get back to your videos and understand the advanced concepts. I always recommend your channel to my friends and one who wants to learn ADF.

[@WafaStudies]

Thank you 🙂

[@shashank.bharadwaj]

Watching after 4 years. Fantastic tutorial. Easy to understand and follow. Kudos!

[@deepayansarkar4258]

This is so far the most simple yet detailed video on ADF CI/CD I have ever seen. Compelled me to subscribe your channel.

[@WafaStudies]

Thank you 🙂

[@kartikeshsaurkar4353]

Thank you so much. This is the only channel which covered Azure Data Factory in comprehensive way. Keep it up !

[@WafaStudies]

Thank you ☺️

[@brainwarmer560]

Well explained. Looking forward to your future Databricks series!

[@joseduarte5663]

Awesome video and very clear explication, actually is the only vid I've found that talks about CI/CD in ADF that is so clearly explained, so please keep posting more vids like this one!

[@amarjeetbatra]

I was struggling with Azure documentation on ADF CI/CD and then found this video. Excellent video explaining how to promote ADF from Dev to other environments using CI/CD.

[@WafaStudies]

Thank you 🙂

[@ddd9675]

One of the best explanation for CI/CD

[@ShwetaDeshmukh-tu3uv]

Thank you for best explanation for CI/CD, appreciate your efforts, Please keep posting such important topics related to Azure Data Engineering, would be more grateful. Keep growing!!

[@santoshonlyforU]

Thank you Mr. Maheer.....U made it so simple to understand what looked like too complicated to me earlier...Feel very confident after watching this video....thanks again

[@WafaStudies]

Welcome 😇

[@sudipmazumdar5358]

Extremely well explained. Thanks a lot for such a detailed discussion on this topic !!

[@WafaStudies]

Thank you 😊

[@ambar752]

Your all Azure Videos are Awesome !!!

[@WafaStudies]

Thank you 🙂

[@azimbaig8556]

Excellent tutorial series. Very useful for those who are new to this world. Explained in very simple language. Thanks Maheer

[@sivakumart8542]

Fantastic explanation; great level of learning. You are much welcomed and appreciated.

[@chad9861]

Thanks for the very detailed explanation of the ADF DEPLOYMENT process..

[@Jai_Radhe_Radhe]

We must appreciate his efforts....Thanks

[@sowmiyas1559]

One of the best topic, thankyou for the wonderfull explaination

[@WafaStudies]

Thank you 😊

[@ShivKumar-ed6el]

Superb Azam, always easy to understand all things. Thanks once again.

[@WafaStudies]

Welcome 😄

[@lathap7391]

Great explanation, really gained higher level of confidence...All your efforts are really appreciated and thanks to keep on encouraging the learners

[@WafaStudies]

Thank you 😊

[@AbdullahAli-yq7wm]

Best tutorial ever. Appreciate it brother

[@bhargavkumar4724]

One of the best Explanation...

[@WafaStudies]

Thank you 😊

[@nidhi1888]

Amazing, I have been following all your videos from the beginning. It is very helpful. Thank you

[@WafaStudies]

Thank you 🙂

[@tipums123]

Best CICD video ever!

[@WafaStudies]

Thank you 🙂

[@Rafian1924]

Great tutorials on adf so far on any channel.

[@WafaStudies]

Thank you 😊

[@vijayadityayadavilli8042]

Excellent Explanation. really Appreciate your efforts. Thanks for making this Video.

[@MrZoomok]

Great tutorial again !!! Thanks WafsStudies team

[@WafaStudies]

Thank you 🙂

[@deepjyotimitra1340]

Very nice.Detailed explanation.Thank you so much.

[@WafaStudies]

Welcome 🤗

[@EuphonicAura]

Good going man. You create extremely Helpful content. Keep it up.

[@WafaStudies]

Thank you 😊

[@sudhirg8773]

Covered all bases and pretty good explanation. Keep up all the good work.

[@VinodKumar-lg3bu]

Very nicely explained thanks Maheer for sharing the knowledge !!

[@manishdubey8646]

Excellent explained in simple way

[@WafaStudies]

Thank you 😊

[@ShivKumar-ed6el]

Thanx Azam for explain, its very usefull...😊

[@KARIMKites-Faisal_Iffat]

Great work Maheer, you are a gem..

[@mustafakamal5945]

Thanks for this video, really well explained, I was struggling to get this concept, now it is clear !!

[@WafaStudies]

Welcome 🤗

[@kulasekharreddy8764]

Again, Thanks for putting all your efforts to make these videos for us. I have been following your ADF videos and finally came to here. I hope this knowledge will help to me in my coming work in Azure.

[@saipraneeth1660]

wonderful bro....as i am completely new to this azure datafactory concepts .. its really nice to understand the concepts :):)

[@WafaStudies]

Thank you 😊

[@jogacharyulu]

This is simply EXCELLENT!!!!

[@WafaStudies]

Thank you 🙂

[@souranwaris142]

Hello, I need help with Delta load (incremental load) from the Oracle server to blob storage(CSV). First I loaded the table with full load by copy activity now I want to make another pipeline with the trigger that run every week load new data and append in the master CSV(that i loaded first pipeline). or you can suggest me how I can load table from oracle to data lake in csv with Delta load (incremental load). I'm attempting to obtain fresh data using a query-based lookup activity, but I have no idea how to use data flow to add the fresh data to the master CSV in the data lake. Perhaps I'm going about this the incorrect way.

[@gagandeepkalra3775]

Will the CI/CD pipeline create the existing folder hierarchy inside ADLS Gen2 account from Dev env to Test and Prod env? And what about the Databricks notebooks, can we get them too in the Test and Prod env.?

[@saipoojithakondapally4136]

Sir, for example I have 3 tasks in adf test and out of three only 1 got qa sign off and not the others. Can you please let me know if there a way to move that one pipeline alone to prod instead of all the three.

[@sandrojorgeoliveira175]

Thank you very much, MAHEER!

[@manjunathbn9513]

Azure Resource Group deployment is not available in the task

[@divyaraghavan9437]

AZURE resource group is missing in my tasks in azure devops.anyone know the reason?

[@srinidhim7978]

I also cannot find it. Use ARM template document.

[@azure1586]

Great.Thank you.Any video on Big data architecture !

[@balajirangu2927]

Thank you bro for providing these useful vedios.

[@saiabhilash3151]

Superb tutorial .Hats off to you patience bro .Thanks you so much for the video .If possible could you also please make a video on the same making new changes in Dev adf pipeline and pushing that new changes to UAT .Also in real time we will not push to UAT and PROD simultaneously .So could you pls show how to push code from UAT to PROD .

[@saurabh3614]

thanks Maheer, great work

[@WafaStudies]

Thank you ☺️

[@muppedaanvesh]

Great explanation!! 💥

[@Soumadip.B]

what if we have to deploy multiple ARM templates as part of release pipeline?

[@somd506]

Fantastic! Problem solved! Thanks

[@arunavaghosh2321]

Great video. IT is really helpful. Thanks Man

[@WafaStudies]

Thank you 😊

[@pradeepsharma1927]

Great Work Sir

[@WafaStudies]

Thank you ☺️

[@ranjitreddy7918]

Awesome explanation

[@WafaStudies]

Thank you 😊

[@ameyraj4947]

This is awesome ❤ loved the video🎉🎉🎉 Thanks a lot

[@WafaStudies]

Thank you 😊

[@santhidhanuskodi8668]

azure resource group deployment task not found in azure dev ops. Is it removed, what is the new one called?

[@arnavverma8622]

The explanation is awesome but the length of the video is so huge that it is impossible to understand it all at once. But thanks for the session Wafa, it helped me a lot. Excellent Course 😊👌👌

[@rajeshmanepalli7367]

Thanks Maheer such as nice explaination

[@naturevoice6535]

Thank you so much, well explained. I have one question, if I add a new pipeline to existing pipelines and trigger the deployment into UAT/Prod, will all my old pipelines get re-deployed every time? if yes, any way to segregate them?

[@soumalyade]

Awesome video... clearly explained. Had to subscribe to your channel for more videos.

[@anishdahiya5811]

@wafastudies plz explain migration in adf and how to deploy adf

[@suresh.suthar.24]

Awesone content. thank you maheer

[@SantoshSingh-ki8bx]

I have created a feature branch but forget to delete it. When again I am developing new changes with this branch and pull into master I am getting merge conflict issue and also don't know how to delete my unused feature branch. Please help

[@vishaljhaveri7565]

Why are we not enabling GIT for UAT and PROD Environment? I am not getting it. 11:00

[@subbumareedu8506]

How to call a foreach activity in another foreach activity is it possible,if not any other way to call? Could you please help me on this.

[@WafaStudies]

Foreach inside foreach is not supported. You use inner foreach as another pipeline and use execute pipeline activity to run that child pipeline

[@arupnaskar3818]

Excellent sir, Thank u for providing great tutorials... Sir, I am stuck in release process where prod adf pipelines are having different triggers(schedule, event based).. And if I do release I need to disable them first.. which I really don't want .. please, give a suggestion to overcome this 💐🍁🧚

[@yassaffi]

Hi bro....I attended one interview where they asked me to design flow diagram for azure end to end flow...could you please let me know where i find good solution for this?

[@sinadandan]

This was great! thank you sir!

[@WafaStudies]

Thank you 😊

[@gopalakrishnasuryadevara1019]

you are making it easy for learners like me to get a good understanding of real life scenarios. Thank you very much brother. Hope your work is going smooth at Microsoft🙂.

[@WafaStudies]

Thank you ☺️

[@CoolGuy]

Azure Key Vault service works for DaltaLake Gen too or only GPV2?

[@azure1586]

How to share a pipeline to another Azure account ? It is possible to download the support files by right clicking the pipeline.But how it can be shred with a friend.So that he can import it and make use in his account ?

[@azure1586]

SSIS : copy the dtsx file and attached it to a target VS studio.Is there any similar method in DF

[@tipsandhacksbygaurav]

really nice effort, can you make a video on automated publish adf using CI/CD pipelines instead of ADF studio publish button.

[@krupa825]

Superb content.

[@marcelaandreacastrillonbui7951]

Hi Wafa, for the variables that changes between the environments where I storage them. By the way amazing material thank you ❤🎉

[@krishnakanthmacherla4431]

how do you do this man , the skill you possess is very 😮

[@namangupta7111]

Very well explanation

[@varunkulkarni7908]

Very Impressive Video, Please create ADF CI/CD using YAML Code as well, that will be great

[@WafaStudies]

Thank you 🙂

[@desparadoking8209]

Thank for Video.🙂☝

[@WafaStudies]

Welcome 😄

[@SantoshSingh-ki8bx]

Thanks for nice video. Here you are deploying ADF only across all environment in CI/CD. Let say we have to deploy all services inside resource group in CI/CD environment. How to do this. As in demo you are stilling some part of manual work for all environment.

[@WafaStudies]

That demo is specific to Azure data factory alone. And that is the way recommend by Microsoft to deploy your data factories to different environments. For other resources we need to research are they support git integration if not may b some custom scripts needs to prepared and trigger them

[@himanshutrivedi4956]

Wonderful video and it will be very helpful

[@minecrafmasterpl]

I have a problem, where the contents of the blob storage do not migrate when deploying the release pipeline. Do you know what this might caused by?

[@sagar_patro]

Excellent video I have watched so far with lots of content. I will request @wafastudies to create a video for HDInsight cluster i.e. How we can deploy code in HDInsight cluster, how can control the versioning in Azure Repos for the HDInsight cluster and creating end to end project pipeline. This will help a lot since this is very new.

[@KrisKoirala]

Well explained, thanks!

[@WafaStudies]

Thank you 🙂

[@bhavnakamble975]

Hello, i was just trying the deployment, but got error saying "Service Connection with that name already exists" while authorize the subscription. Do i need some user permission to use this existing connection?

[@kislaysinha9772]

thank you WafaStudies

[@WafaStudies]

Welcome 😊

[@murugeswarimuthurajan538]

Is it possible when we have many pipelines, datasets and linked services in our Azure Data Factory?

[@WafaStudies]

Yes it will work

[@murugeswarimuthurajan538]

@@WafaStudies Thank you 🙂

[@kiranpawar451]

Can we do ADf testing in Devops & how to do it?& Which test framework use?

[@ItsmeRajivS]

Nice Video bro.. is there any video on automate publish in ADF with CI/CD pipeline

[@SaurabRao]

This deployment is the classic deployment model, correct? Do you have recommendation for a similar tutorial on YAML deployment?

[@WafaStudies]

Sorry Saurab at this point, no i dont have. Try search in google or KZbin

[@SaurabRao]

@@WafaStudies no problem, thank you for your reply!

[@anupgupta5781]

Hi Can you help with full Automated approach as well for this we need to click on publish button in master branch

[@VinayKumar-st9iq]

Hi Wafa. In Azure Devops iam not able see the release pipeline in pipelines section. can you please help me to solve this?

[@TheAbhisai]

go to organization settings and then you have unclick the disable option for release pipelines then it will be visible

[@mohammadrafi2902]

Excellent and nice information. Thank you. I have one question. Need to create resources in another environment UAT and prod whatever have in dev environment like blob storage, Adf, Azure sql while deployment?

[@theenduser1037]

Hello Mahi, can you help a video with cicd with size more than 10 mb?

[@Rani-le1go]

awesome content

[@moviesreview3539]

Can’t we move a 5 pipelines out of 20 in cd release

[@shaiksamad3143]

Hello thanks for the video . I have one issue i got everything deployed in production environment but not the files in storage account of production resources group

[@Ravi-gu5ww]

Is it way to deploy in synapse also or any other way

[@ronakagarwal8529]

Can you please share the video of CI/CD in Azure Databricks.

[@bhavnakamble975]

Just like your other videos, this one was also quite informative. I just have one question here. After making some changes in dev in future, do we have to create a new release pipeline or can the existing one be used?

[@WafaStudies]

Existing one will work if u have selected incremental load inside your release pipeline

[@bhavnakamble975]

@@WafaStudies understood. Thank you!!

[@kishorekumar-lo9iy]

Hi sir, what's the purpose of stopping triggers in azure data factory? Can you plz explain me in real time scenario?