Deploying Azure Data Factory With CI/CD Using Azure Pipeline

[@empowerpeoplebytech2347]

Fantastic! Thank you very much for this detailed video on ADF CICD. I was stuck at an initial point where my account and project were not visible in ADF, it helped. Also, now have a clear understanding how the flow from various branches takes place. This is a very big thing which every ADF developer must know. Thank you very much. Wish you lots of success!

[@MRadwanMSF]

You're very welcome!

[@msvasankv]

It's a great presentation and a nice walkthrough on creating all the stages. Thanks, Mohammed!

[@MRadwanMSF]

Glad it helped! and thanks for your comment!

[@riojedac]

Good to find you here Mo! This is a great and didactic video, thanks!

[@MRadwanMSF]

Glad you liked it!

[@jasonbougas2999]

Thanks for this detailed walk through. I got up and running with my Dev to UAT deployment really fast following your demo. Really helpful!

[@MRadwanMSF]

Glad it was helpful!

[@castrojona82]

Muchas gracias. Años y años con este dispositivo guardado, me salvaste de una compra mas.

[@irammapatil9957]

This is really helping for me...i got so many things which i was looking for..Thanks

[@lukeroberts70]

Fantastic walkthrough, helped a lot, thank you so much!! One thing I did need to do for the ci-cd.ps1 to run was under Use Latest PowerShell Version, was to choose Use PowerShell Core under the advanced option below. Otherwise it didn't like the format of the script.

[@saikarthik4a9]

Thanks Radwan, It's a clear explanation. Looking forward from you to get more videos

[@MRadwanMSF]

Thanks!

[@alphacharith]

Thank You Mohamed Radwan. Excellent Video

[@MRadwanMSF]

Glad you liked it

[@bharathsairam443]

Fantastic...Very good explanation...Thank you Mohamed..

[@MRadwanMSF]

Glad you liked it Thanks for your comment!

[@bonnevilleedits2303]

There's a slight mistake in the values for Start and Stop Trigger. In the Video Mohamed sets the delete deployment flag to true for clean Resources and Start Trigger. But you also need to set the predeployment flag to false for it to start the triggers again as this is the flag to distinguish between pre and post deployment.

[@MarimuthuSA]

Great Video; Detailed explanation. It helped a lot.

[@MRadwanMSF]

Thanks Mari, I am happy you like it!

[@HemantKrSahu]

Bravo Man !!! Thanks for making this helpful video.

[@sriramudhyab]

Excellen video and very thorough explanation..Thank you

[@MRadwanMSF]

Glad it was helpful! Thanks for your comment!

[@lifechamp007]

Super helpful - Thank you so much Radwan !! #StayBlessednHappy

[@syedfarhaan-e9k]

thank you so much bro. bless you bro for this! this is really helpful.

[@minineng21494]

Thanks so much for the video, really helpful!!

[@Mramaamo]

Thanks Mohammed, its a very good one

[@MRadwanMSF]

Glad you liked it! Thanks!

[@terryliu3635]

Thanks! This is what I’m looking for.

[@MRadwanMSF]

Glad I could help!

[@NSys-ru3ix]

Excellent tutorial, thank you.

[@UmerPKgrw]

Very Well Explained

[@sanjithkumar7600]

thanks for the end to end demo :)

[@CoopmanGreg]

Excellent Video! Thanks!

[@MRadwanMSF]

Glad you liked it!

[@SynonAnon-vi1ql]

31:29 You see everything is great in this video but I am not understanding one thing. I would really appreciate if you could please explain. My question is why did you merge the code from feature branch to master without first testing it in the UAT environment. This is opposite for me. I was thinking you will show the automated mechanism to first merge the dev code to "UAT" branch, then deploy it in UAT env for testing purposes. Once the testing is successful, the code from UAT branch will be merged into Master branch and from there it will be deployed into production. But you first merged the Dev code into master without verification (UAT). Now if the UAT fails your master is no more good and is not representative of the prod. Can you please explain why my thought process is wrong? Thank you!

[@kjbiliks]

This information is very informative and has been incredibly helpful, especially due to its hands-on nature. Instead of creating a single resource group for all three environments (dev, UAT, and prod), I believe it is more practical to create separate resource groups for each environment. This approach better separates the environments. What do you think?

[@gustavoortega4361]

Thanks for the video :)

[@MRadwanMSF]

My pleasure!

[@LandscapeInMotion]

Thanks for not speeding through this demo!

[@gobindamohan]

This is really awesome

[@MRadwanMSF]

Thanks!

[@abidnadia6382]

Thank you for sharing this relavent information however I would like to ask about data flow and where to make the data transformations ? Thank you in advance :)

[@minufrancis222]

I see a task - publish data factory - is this the new form of deployment or not? It has checkboxes which says 'StartStopTriggers' as well

[@prabhurathinam]

Hi Mohamed, Thanks for a great video. Appriciate it. One of my linked service is conencted to Dataverse. In the template parameter i'm unbale to pass Service uri for the uppper environements. i can only able to pass Service connection and its secrect. is there a way to pass the Service URI as well in the parameters? Please advice. Thank you

[@swapnilwagh7515]

Nice one!

[@MRadwanMSF]

Thanks!

[@sjitghosh]

Excellent

[@MRadwanMSF]

Thank you so much 😀

[@galilioXX]

Thanks a lot Mohamed for the great presentation, very informative and straight to the point! I configured the CICD pipelines for my organization to auto-deploy ADF changes, however I ran into an issue in which the deployment messed up my scheduled triggers start times as it used the start times from the source code which is an old value. I was wondering if there is a way to deploy all the changes without changing the triggers start times? if no, how we can keep the trigger's start time value from being overwritten?

[@arupnaskar3818]

Hi Nemairi, Do u know is there any way to release pipeline with disabling adf triggers.. ?? Please, I am stuck 😔

[@bitips]

Thanks for that, But in different environments we have different DBs (connection strings), so how can we manage this dynamically?

[@sabirpatel3555]

Thanks for the video, Mohamad Radwan , Can you please create a video or guide us how to create a CI-CD pipeline using Azure SQL table. how should we proceed for that.

[@amitsinhaism]

but how we would change the storage accounts for UAT and prod as they would be different for UAT and prod ?

[@DINESHKUMAR-cu2db]

Unable to connect the AAD in Azure devops. Error is the current organization policy does not allow guest users to access the organization . Change the policy setting to external guest access

[@kamblekiran786]

Very nice video @mohamed, Can you please elborate more on powersell script we are using it. For what purpose it is used?

[@AndersKnudsen93]

Thanks for the great video, Mohamed! I was wondering how this would work if the ADF kept on growing and more and more pipelines along with more linked services would be added. Would that require to keep adding connection strings and alike to the "override template parameters" section of the release pipeline?

[@debbieedwards7267]

So helpful. Thank you. My only problem is that my dev and prod are in different subscriptions and resource groups so I don't quite know how to tie this into your where your dev and prod are in the same subscription. for example. At 40.51 when setting up the ARM template. Should the resource group be your dev or Prod one if they are different?

[@MRadwanMSF]

Glad it was helpful! This is just a small example but in reality you should parameterize everything and most probably we will use variable group docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml

[@AxL28AxL]

For the Data Factory Scheduled Trigger, in my Dev the Trigger status is Stop and I would like to ask if I deploy it to Test or Prod would the "Stop Trigger" and "Clean Resource Resource and Start " task would start the status of the Data Factory Scheduled Triggers?

[@MRadwanMSF]

The script should start all triggers

[@HanumanSagar]

Hi can you help me in stop trigger task while selecting Azure subscription I'm not able to see the subscription list..what should I do if want to see list of subscriptions?

[@camassasantiago933]

great video, I wonder can we just get changed files in repository and only deploy these changed files without all files. I stuck in how to do that

[@MRadwanMSF]

You can use another way that only deploy the ADF pipelines that you want, see this link: marketplace.visualstudio.com/items?itemName=liprec.vsts-publish-adf

[@rameshbaddam5465]

I just have a question . Where is the build pipeline ?. Don’t we have build pipeline like other services for adf?

[@kannan114]

What's the difference between adf and master branch? Is master a staging and adf actual master in each environment?

[@ravindranathmetla8180]

Mohammed, having issues to deploy data factory with global parameters.. can you send any link or info regarding this..

[@ketanshah9082]

Awesome work Mohamed, you explained it so well in your video. thank you so much. Mohammed is there anyway you can give access to this presentation(ppt) you prepared for this demo?

[@MRadwanMSF]

Thanks, I am happy it was helpful. Sorry, I can't find the ppt.

[@ketanshah9082]

@@MRadwanMSF no worries Mohammed. Please keep up this great work you do in teaching some of us so easily and with step by step guidance. I am looking forward for Azure DevOps with Databricks, Synpase, HDInsights would be great to learn.

[@adamkhouiy2719]

Excellent video Mohamed. Thank you!!. I have just some questions about deployment in different resource group (Dev, Uat, prod) and each time with different resources names. For example using dev-storeaccount in Dev resource group and uat-storeaccount in Uat resource group and so on ... My first question is : do we use the target resource groupe name in start/stop trigger script ? . Second question how we can handle the parametrization of resource name from an environement to an other ?

[@MRadwanMSF]

Thanks! You should parameterize everything and most probably we will use variable group docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml

[@adamkhouiy2719]

@@MRadwanMSF Thanks !

[@harishchalla268]

Thanks for the great video, Mohamed Radwan. The pipelines which you are pushing to the destination(UAT), it reaches to UAT live mode. Is it possible to move pipelines from DEV to UAT git repo mode instead of livemode ? (in which both DEV and UAT are connected to git.)

[@chennareddy2770]

Thanks Mohamed, this is really great. I have a question on stop and start trigger , if i dont include the script which you have added before and after the ARM template deployment, will it just copy the datasets and pipelines but will not trigger the pipeline to copy the blob content to source to destination correct? I have multiple Data Factories in my resource group and is it safe to use the stop and start trigger script, hope that it will not disturb the other Data Factories

[@Andyfshito]

Hi! I would like to know if there's a way to avoid clicking on Publish every time you perform a pull request to the master branch. Is that possible? Thank you!

[@gameimdum7396]

Thanks mate

[@MRadwanMSF]

Happy to help

[@amitghotikar]

Predeployment should be set to false for starting triggers , isnt it?

[@antongolubev3511]

Can I use multiple ADF for developing one pipeline in one repository?

[@abreezy1st]

what do we do if we have copy pipelines in dev and uat but we dont want them in prod? is there a way to specify which environments to pass copy pipelines into?

[@colinrippey1723]

Great presentation. In a scenario where you have say two developers and each developer works on their own dev branch, would you have two ADF-DEV instances or would the two developers share the single ADF-DEV instance and work on their individual branches via the git integration? I am trying to get my head around what happens in Data Factory as you switch git branches, does all the associated artefacts “swap themselves out”? Does the Data Factory git integration really imply you just have one ADF-DEV and one git branch being developed at a time? Any thoughts you have on this would be most appreciated, thank you again for taking the time to put together such a useful video.

[@bhavyavful]

Hi Mohamed. This video was great. I have a question for you regarding ADF. I am facing a problem in creating a single pipeline. Pls reply if you are interested to sort out my problem.

[@rodrigueschetan]

Hi Radwan , Can we parameterize self hosted run time in Azure data factory while doing azure devops ?

[@tipums123]

Thanks Mohamed, it was a great presentation. One question - Instead of Master can we add Powershell script files to adf_publish branch? Is there any pros or cons adding them in adf_publish?

[@MRadwanMSF]

Thanks!

[@andychiarwantopranata6579]

Hi, Radwan. Thanks for the detailed explanation! I wonder if we can change the dataset source programmatically in release pipeline? For example: in ADF-Dev I use Azure SQL Database as dataset source, but when publishing to ADF-UAT it automatically changed to SQL Server On-Premise Database (assuming both of data source has the same table structure).

[@MRadwanMSF]

Yes you can do that, you can use YAML and replace function. See the following. devblogs.microsoft.com/scripting/use-powershell-to-replace-text-in-strings

[@andychiarwantopranata6579]

@@MRadwanMSF Thanks!

[@bharanidharanm2653]

Hi Mohammed, Thanks for the video. I am using managed identities to blob storage and database. How can i form the variables for "override template parameters" could you pls help clarifying. thanks.

[@abhijnanbp1397]

Hi Radwan, This is a great video! I have a question - in this video, you have demonstrated that all those changes made in Dev will be deployed on UAT and Production in the same CD pipeline. However, in the real time scenario, if I make 10 changes in Dev, all those 10 changes should be deployed on UAT. Imagine, out of those 10, we got an approval to deploy only 4 changes onto Production, and the rest of the 6 should still be in UAT for extended testing. I am curious to understand how can we deploy only those 4 changes onto production. I look forward to having answer, and that will be of great help. Thanks in advance!

[@TheMadhan1988]

Did you get a solution for this?

[@BharathKumar-zq6lm]

Great presentation Sir I have one question - we have 3 different self hosted integration runtime with different names for each environment dev, uat and prod respectively. how to parameterize it while deploying to uat and prod env?

[@rodrigueschetan]

Hi Bharath, did you parameterize self hosted IR now ?

[@hamzamanouzi4937]

Hi, Radwan. Thank you very much for this! It saved me a lot of time. I have one little question: what if the data factorys are in two different resource groups? Thank you very much again.

[@shwetasaronja2110]

We need the powershell script that you copied from microsoft website at the time of creating a file.On microsoft website it is not available. Please watch video at 34.26 time and guide me

[@jubatistim]

Great presentation man, thank you! Where I can get the ci-cd.ps1 file content?

[@MRadwanMSF]

I am happy it helped. Here is the link where you can find the file docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment

[@hemanthkh]

Please share the yaml pipeline

[@snad256]

Hi I'm unable to add custom parameters for Snowflake. Could you please get back to me?! Great video man! Thanks

[@AxL28AxL]

So the UAT and Prod don't need to be associated with Azure DevOps Git?

[@MRadwanMSF]

Yes, only Dev assoicated with Git and the rest of envs are not

[@AhmedEssam-m9y]

هل ممكن حضرتك تعملنا فيديو بالعربي شرح ل azure data factory

[@MrDwikipedia]

Thanks for this it's been very helpful! I got it working after a lot of trial and error. One question and maybe I'm misunderstanding, but shouldn't the start trigger be -predeployment $false ?

[@MRadwanMSF]

Glad it helped! Yes, for the start trigger and if I didn't do that, it's my mistake :-)

[@irammapatil9957]

Hi, can you send powershell script link which you have copied from microsoft website

[@RajeevKumar-zf8ox]

Hi Radwan, I have one Databricks. How to deploy it using CI/CD

[@MRadwanMSF]

You need to use this tasks marketplace.visualstudio.com/items?itemName=riserrad.azdo-databricks

[@NitinMathewGeorge]

But now the arm approach is having limitations with parameter limit of 256. Secondly the concept of branches in this approach isn't ideal and acceptable from MS.

[@krishnachaithanya2600]

too frequent shuffling during 'merge changes' section

[@zabeehullah3673]

​Hi, I have a scenario and need community advise to manage it in best possible way. Due to compliance issues, I am not allowed to save our source code in any Geo location supported by Azure Devops Services. so under repo when we create a repo, by default its created somewhere in Geo location, ideally where azure devops service is hosted.But this approach is NOT-OK for us. And I want to use our own server "hosted in MS-azure" as repo server.so whenever, we create a repo using create repo under the project repositories this should be stored in that particular server.. please advise how can i achieve this? our own server is having Visual studio 2019 and connected to azure devops also.

[@shakechen7944]

you can use tfs server deploy local.

[@mohammedalbosifi3975]

بدنا دورات بالعربي

[@MRadwanMSF]

احضر المحاضرة الجاية kzbin.info/www/bejne/mnW9qGSBbNSqbJo

[@saisylaka955]

Hi Mohamed, I have follwed the same setps which you explied in the video. I'm getting below error while running the pipeline. Please help me out in this. Error: 2023-11-03T06:31:42.8563878Z ##[error]Could not find the modules: 'Az.Accounts' with Version: ''. If the module was recently installed, retry after restarting the Azure Pipelines task agent. 2023-11-03T06:31:42.9672510Z ##[error]PowerShell exited with code '1'. 2023-11-03T06:31:45.6177222Z ##[command]Disconnect-AzureRmAccount -Scope Process -ErrorAction Stop 2023-11-03T06:31:45.7413326Z ##[warning]Run Connect-AzureRmAccount to login.