17. Databricks & Pyspark: Azure Data Lake Storage Integration with Databricks

[@JalindarVarpe-h4o]

Enjoying the PySpark tutorials! Can you make a video on setting up Azure and navigating the portal? It would be super helpful. Thanks for the great content!

[@a2zhi976]

you are my guru from onwards ..

[@rajasdataengineering7585]

Thank you

[@shivanisaini2076]

this video is worth watching, my concepts related to access the file in databricks are clear now thank you sir

[@rajasdataengineering7585]

Thanks Shivani!

[@ndbweurt34485]

very clear explaination. god bless u.

[@rajasdataengineering7585]

Thank you

[@HariprasanthSenthilkumar]

Can you please make a video to connect to ADLS by service principal

[@rajasdataengineering7585]

Sure I will do

[@dhivakarb-ds9mi]

I am getting this error Operation failed: "This request is not authorized to perform this operation using this permission."

[@natarajbeelagi569]

How to hide access keys?

[@rajasdataengineering7585]

We can use databricks scoped credentials

[@alexfernandodossantossilva4785]

If we have a Vnet on the storage account? How can we access?

[@naveenkumarsingh3829]

hey you are using location as wasbs:// which is nothing but azure blob storage location , and sometimes you are taking abfss:// which is path to azure data lake gen2 location.. Since I am still learning , I am getting really confused now.. And your video says adls connection with databricks..then it should be abfss:// right for a file path?

[@rambevara5702]

Don't we need to app registration for data lake?

[@rajasdataengineering7585]

That is another way of integration through service principal

[@rambevara5702]

@@rajasdataengineering7585 whatever it is fine right..brother where can I get this databricks notebook..do you have any GitHub

[@felipedonosotapia]

Thanks so much!!! nice tutorial

[@rajasdataengineering7585]

Glad it was helpful!

[@lucaslira5]

How would I do if the container had more files instead of just 1?

[@rajasdataengineering7585]

We can use wildcard to select multiple files

[@lucaslira5]

@@rajasdataengineering7585 what would this wildcard be like? I have two files in the container (city.csv and people.csv) but it's only bringing people.csv

[@rajasdataengineering7585]

You can give *.csv so that it can pick all CSV files

[@lucaslira5]

@@rajasdataengineering7585 But I would like to bring a specific file, for example my blob has 50 .csv files but I only want to bring people.csv to perform an ETL

[@lucaslira5]

Would it be here for example to put .option("name","people.csv)? df = spark.read.format("csv").option("inferSchema","true").option("header", "true").option("delimiter",";").option("encoding","UTF-8").load(file_location)

[@Ramakrishna410]

Great knowledge. How can we apply access polices on mounted containers? For ex , 50 users have acess for databricks so , 50 users can see the all files under mounted container but i want to give read acess for few users only? How can we?

[@rajasdataengineering7585]

Hi Alavala, Good question. Mount points can be accessed from darabricks through service principal or Azure Active Directory. If we use service principal (SP) to create a mount point, all users/groups under the databricks workspace can access all files/folders in mount point. So if you want restrict access for set of people, there are many ways. One common approach is use AAD to create mount point so that user access ca be controlled using IAM within Azure portal. Another approach could be creating 2 different databricks workspaces and accessing mount point through 2 different service principals one with read access, another with write access. Hope it helps

[@sujitunim]

Really very helpful... could you please create video for on premise Kafka integration with databricks

[@rajasdataengineering7585]

Sure Sujit, will do one video on this requirement

[@rajivkashyap2816]

Hi sir, Any git link is dere so that we can copy and paste the code

[@jagadeeswaran330]

Nice explanation!

[@rajasdataengineering7585]

Glad it was helpful! Thanks

[@lucaslira5]

with this option, is possible writing in data lake? Or only read?

[@rajasdataengineering7585]

We can write as well

[@dataengineerazure2983]

@@rajasdataengineering7585 How get the dataset source(csv files)? thanks

[@AkashVerma-o7o]

is it free to use azure data lake?

[@rajasdataengineering7585]

No, it's not free

[@kartechindustries3069]

Sir does azure data lake comes under community groups or free services

[@rajasdataengineering7585]

No, azure data lake is paid services but Microsoft provides one month free subscription with some free credit. You can take advantage of it for your learning purpose

[@DivyenduJ]

Hello All, I am new to this and getting below error , many thanks if anyone could help for step 1: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

[@rajasdataengineering7585]

Hi, seems the access key is invalid. Could you check it once again from storage account

[@DivyenduJ]

@@rajasdataengineering7585 Thanks a lot sir for the guidance it worked , mistakenly set on rotate key. May be that's the reason.

[@rajasdataengineering7585]

Glad to know it worked!

[@subbareddybhavanam5829]

Hi Raj, Can you please add data files too. like CSV and Json ...

[@MohanGonnabathula]

Yes

[@sravankumar1767]

Nice explanation bro 👍

[@rajasdataengineering7585]

Thanks bro

[@anoopkumar-f1r]

Great Raja!

[@rajasdataengineering7585]

Thank you

[@lovepeace2112]

good