18. Databricks & Pyspark: Ingest Data from Azure SQL Database

[@venkataram6460]

simple but effective process and expalination as well. Great job

[@rajasdataengineering7585]

Glad you liked it!

[@omprakashreddy4230]

Simply Awesome !! We are learning a lot from your videos.

[@rajasdataengineering7585]

Thanks Omprakash

[@avinashpasupuleti07]

Simply Super Bro.. Awaiting for more videos from you on db related activates in data bricks

[@rajasdataengineering7585]

Hi Avinash, thank you. Sure will post more videos on db related activities

[@Khm3rick]

Great video! Just one question, I saw you defined the jdbcDriver...but I didn't see it used after in jdbcUrl? What is it for?

[@ShivaKumar-dj8bj]

Yes I have also observed it. Later in other videos I found out that we have to add it as an option when reading data into data frame like .option("driver", jdbcDriver")

[@sushilkushwaha260]

Awesome, Very nice explanation...

[@rajasdataengineering7585]

Thank you

[@sravankumar1767]

Nice explanation bro 👍

[@prathapganesh7021]

I really appreciate this video thank you🙏

[@rajasdataengineering7585]

Thanks Prathap! Glad you find it useful

[@prathapganesh7021]

Can i join for paid project.. Or else how can i contact you..

[@kylebrogan6416]

What about just being able to access those same tables via an existing databricks catalog in the hive metastore structure for example? Is there a way to do that?

[@dataisfun4964]

Thanks worked perfectly.

[@rajasdataengineering7585]

Great

[@VirajithaPanguluri]

Can we mention type of authentication while connecting? What if we have only Azure Active Directory Password ? How to mention that?

[@a2zhi976]

like in unix , can i save all these details in one file and call in the beginning of the scripts. ?.

[@rajasdataengineering7585]

Yes we can use yaml or json configuration file to save the details and during run time, spark can read the configuration file and process accordingly

[@alwalravi]

How to write this product table data into blob storage in parquet format in a databrick notebook? Plz help

[@rajasdataengineering7585]

We can use databricks writer df.write.format("parquet").save(location)

[@rajasekharmedia8987]

I want to do query from sql and load the result into one variable. Can we do that. Like select max(id) from sql table. I am using this id for comparison in next steps

[@veerag9426]

Super nice video

[@rajasdataengineering7585]

Thank you

[@arnabsontu6578]

Sir, Can we also create , view, alter , run stored procedures from databricks ?

[@rajasdataengineering7585]

Hi Arnab, stored procedures can't be created in databricks. Views can be created and can be altered as well

[@betterahyosi]

You didn't use the jdbcDriver . What is the purpose to have jdbcDriver ????

[@rajasdataengineering7585]

Why do you say I didn't use jdbc driver???? Look at 7:30 in the video

[@betterahyosi]

​@@rajasdataengineering7585I meant that u didn't pass the jdbcDriver value in to the jdbc url

[@shashikantchaturvedi1559]

Hi Raja's, I am following this tutorial step by step but I got an error while running the 2nd cell of getting product table. the error is " java.sql.SQLException: No suitable driver", can you please help in this case.

[@shashikantchaturvedi1559]

Now I got that, something was wrong in preparing the connection. I can connect, and get the data from Azure Sql Server.. Thanks Raja.

[@rajasdataengineering7585]

Glad to hear you fixed the issue 👍🏻

[@vaddenata6735]

Thank you so much ❤Sir....

[@rajasdataengineering7585]

Most welcome! Hope you find it useful

[@vaddenata6735]

@@rajasdataengineering7585 Yes 💯

[@bhumikalalchandani321]

Sir getting no suitable driver on running df ​@@rajasdataengineering7585

[@AjithKumar-cj7hh]

What if data is huge like 100 gb. Is it still recommended?

[@rajasdataengineering7585]

Jdbc connection has performance issue while handling huge amount of data. But there are options to improve the performance which can be applied depending on the use case

[@suresh.suthar.24]

great video

[@rajasdataengineering7585]

Glad you enjoyed it

[@shivangishingatwar1356]

Could you help me , establish connection string using azure active directory authentication mode

[@apoorvsrivastava7121]

Sir how can we connect using serects from keyvault ?

[@rajasdataengineering7585]

We need to create scoped credentials in databricks first to setup integration between key vault and databricks

[@apoorvsrivastava7121]

@@rajasdataengineering7585 thank you will check and do 💪

[@melvin9993]

Simple and effective

[@suman_vedakshaB]

HELLO SIR..WHILE IMPORTING DATA HOW WE COME KNOW WHICH IS MODIFIED AND WHICH IS LATEST DATA??I MEAN ANY UPDATED DATA HOW WE HANDLE THAT..PLS REPLY

[@Jayalakshmi-r9t]

how to get that ip address . i did not find while logging. please can you say

[@rajasdataengineering7585]

You can get it from command prompt using ipconfig command

[@natarajbeelagi569]

I want to connect to my localhost but the connection is getting refused. Can you please make a video on it?

[@arnabsontu6578]

Sir, is there any way to hide the password from exposing it in the code ?

[@rajasdataengineering7585]

Yes Arnab, we can use azure key vault

[@akashsharma4769]

Also we can use Databricks secret scope

[@rajasdataengineering7585]

Yes we can use databricks secret scope

[@Jayalakshmi-r9t]

how to get that IP address , foe me it was not visible

[@SantoshKumar-yr2md]

how to get multiple tables from Azure SQL into databricks notebook

[@govardhanbola1195]

Hard coding Password in the code is not recommended. Can we get password from Azure Key Valt. Can you please let us know the steps for that

[@rajasdataengineering7585]

We need to integrate azure key vault with databricks by creating secret scope

[@MrTejasreddy]

simple superb can you make a video how to creatra account in databricks community addition for free