Azure Data Factory - Accessing a Databricks Notebook with Input and Output Parameters

[@camilogutierrez5491]

Excellent. Best tutorial about the topic

[@debayandas1128]

Thanks @Dinesh for this tutorial. Exactly what I needed.

[@sureshpathak2008]

Thanks for Nice & crisp Video related to Objective you set in opening Note of Video.

[@priyadarshanmishra6687]

Thanks Dinesh sir for this vedio. Can you please tell us how can we directly insert the transformed record from the Databricks into the Azure SQL DB & Azure Synapse.

[@dpatro7245]

If we want to return multiple output value from notebook then how do we get individual value into pipeline, can you please help me

[@GurvinderSingh-lf5xj]

@Dinesh: great demo buddy. ONE CONCERN- how to pass the output file content into database?

[@pranavkumar9030]

Hi Dinesh your videos are super good, if you can help me what will be the typical use case when we can use data bricks inside azure datafactory since most of the transformations can be done in adf itself.

[@prajaktaakhegaonkar8674]

Very helpful. This cleared may things for my requirement.

[@corsaronero5619]

Great Dinesh, i have a question, i have an ADF pipeline which use web activity to call an API, it returns a NDJSON that I need to convert in JSON, so I was wondering to use databricks, but I can I pass the web output to DB? thanks for your reply. Corsaro

[@DineshPriyankara]

I have not tried out Nero but I think it is possible to pass NDJSON (as a text) to Databricks and then get it converted or processed.

[@amit09001]

How do I run a scheduled job with different arguments each time in databricks without ADF ?

[@saikatsengupta6675]

Very well explained. Great content. Can we get the data you referring over here?

[@TheVijaynegi]

it really to the point and helpful

[@dipanjansaha6824]

Great article 👌 and we'll explained. A question: we can do the same thing using databricks inbuilt job functionality right?

[@DineshPriyankara]

Hi Dipanjan, Thanks for the feedback. What you exactly referred with "Databricks inbuilt job functionality". With this video, Im trying to pass values from an external party (in this case, ADF), if you can explain this bit more, I will be able to explore it and see.

[@HafizfromIndia]

Excellent Tutorial 👏

[@pankajverma29007]

Excellent ! Thanks for sharing.

[@abhishekv7171]

Thanks Dinesh, this is really helpful

[@vipinkumarjha5587]

I have one query on it, why we have 2 entry in the table for massage

[@shashidasari2]

its from databricks Notebook try catch block 1st time execution it creates the folder path if doesnt exists 2nd time execution the folder exists hence we get message folder exists and updated

[@anoopchaudhary2826]

Very Very Informative Video

[@tannajay90]

Thanks for sharing.

[@arnabdutta3578]

Thank you

[@aks541]

I got the following error when calling Databricks notepad from ADF "errorCode": "3204", "message": "Databricks execution failed with error message: . Run page url: westeurope.azuredatabricks.net/?o=4661525133631160#job/3/run/1.", "failureType": "UserError", "target": "Notebook1", "details": [] }

[@DineshPriyankara]

Hi Ak Sorry for not replying immediately, are you still experiencing the same, not sure the exact reason but generally we see this when the cluster is not available. If you can give me more details, I will be able to check and see.

[@krish808]

Good

[@mahammadshoyab9717]

Hi priyankara, May I have your gmail to send some questions on databricks.