Working with OAuth 2 0 APIs in Azure Data Factory

[@HybridVirtualGroup]

Use the following timings to jump to your content of interest: 00:00 Introduction 02:22 What is OAuth 03:44 The Authorization Flow 11:20 Testing APIs (with Postman) 36:08 Azure Data Factory - Linked Services, Datasets & Pipelines 48:09 Questions

[@JohnOC4U]

Posts such as this our the lifeblood of data engineers!! People such as Martin should be congratulated for going to the trouble to share this type of content! Brilliantly explained. Incredibly useful. Thank you.

[@HybridVirtualGroup]

Thank you for your kind feedback. We agree, Martin is a superstar.

[@MartinSchoombee]

Thank you very much for the kind words, John. Comments like these help motivate us to keep going, knowing that we were able to help some folks.

[@mohanpanga7794]

This is amazing, I liked how you used the Azure Key Vault to securely save the token/url etc to use for the API calls.

[@sallieworsley2867]

Thank you for putting this together, it is informative and useful.

[@eziola]

Great video! So helpful, and great job explaining everything. That being said, I believe you forgot the step where you have to add an "Access Policy" that includes the Key Permissions for the Data Factory.

[@MartinSchoombee]

Yes, good catch thank you :-)

[@DataWithWill]

What exactly is meant by what you said? I am getting an error that says "AKV10000: Request is missing a Bearer or PoP token." which I think is related to creating an access policy; however, when I click on Access Policy it says that everything is handled in the Access Control (IAM) page

[@Ravichandra-fi5ks]

I hope you keep on doing what you are doing. 🙂

[@SanderSchmeitz]

Awesome tutorial! I got my OAUTH2 flow working as well! Only question that I have is: What Sink did you configure in the 'Copy Xero Invoices' copy activity? In my case I want to send data from system A to system B, but I think I just have to use two Web activities (Get and Post/Put) instead of a Copy activity (because I think they don't support POST/PUT in the Sink of a Copy Activity).

[@geoffreyhibon2651]

Amazing Pres! Thanks a lot

[@alikh1984]

Great Stuff, thank you for this!

[@avinashkale5019]

Great video and very nicely explained. I liked to understand how you used the Azure Key Vault to securely save the token/url etc to use for the API calls or is any code publish in Git Repo, if yes please share the link. Thanks in advance.

[@JMo268]

So is postman or other third party tool required to configure Oath between ADF and your example cloud app?

[@viveknimmagadda2397]

Thank you for the detailed explanation. I had followed all the steps as explained here. However, when I create the web activity in ADF to refresh the Xero access token, I keep getting the error "invalid_client" with error code 2108. It works perfectly fine when in Postman and when I look into the input of the web activity, the Authorization and grant_type are exactly in sync with that of Postman. Tried searching for the same error on the internet but wasn't successful.

[@tbuck51]

Great video! Question. I see that the refresh token is updated in Key Vault with a PUT operation, where does the Access Token get saved? Does it get saved under a secret in key vault. I am working on a Quickbooks Online integration that requires a "client secret" and "refresh token" in the linked service

[@MartinSchoombee]

Thank you! Yes, I like to save the access token, refresh token and client secret all in Key Vault. The access token is saved in a similar fashion to the refresh token. Because of the fact that the client secret doesn't change, you can add that to the Key Vault manually and don't have to do it as part of the pipeline. A word of caution regarding the QuickBooks linked service, as I am dealing with it on another project...don't use it. Use the native REST linked service for REST APIs...the built-in linked services usually don't do a good job of dealing with the nuances the different vendors like to implement.

[@tbuck51]

@@MartinSchoombee Thank you for the heads up! Its always a risk when trying to use Preview features as it is. I will try out the REST API connector instead. I appreciate all of your help and great explanations!

[@tbuck51]

@@MartinSchoombee Is there any chance that you may have a tutorial of the QuickBooks Online integration that you are working on as well?

[@MartinSchoombee]

@@tbuck51 Not specific to QuickBooks, but have a look at my blog series on the topic...the logic should be virtually identical to what I have there.

[@tbuck51]

@@MartinSchoombee Will do! Thank you!

[@ris9hi]

I see, you have implemented refresh grant type in ADF, do I need to run the Auth code flow in Postman first and get the refresh token and use the same refresh token in ADF with grant type as refresh?

[@MartinSchoombee]

Yes, you need to go through the Auth flow outside of ADF first in order to get your first set of tokens. After that, you can use those tokens in ADF to get a new refresh token.

[@SachinGupta-nh5vy]

where will i get Auth URL and Access Token URL? please help

[@saurabhtalele1537]

cool content....

[@rashane1000]

new sub here! thanks for this awesome content! keep it coming 😊