05. Databricks | Pyspark: Cluster Deployment

#DatabricksCluster, #Clusterdeployment, #Sparkcluster,
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@ashutoshjadhav6922]

Sir Your videos are really very helpful, nobody has given such information in simple language in their video. Keep up the good work of spreading the knowledge❤

[@rajasdataengineering7585]

Thank you, Ashutosh!

[@anirudhnegi4523]

Great content. Smooth explaination. Keep up the good work!

[@rajasdataengineering7585]

Glad you liked it!

[@saivamsiy7312]

Raja the great 👍

[@rajasdataengineering7585]

Thank you

[@sravankumar1767]

Superb Raja

[@VISHVABATTULA-p7l]

thank you sir

[@rajasdataengineering7585]

Most welcome!

[@kamalbhallachd]

Helpful and knowledgeable session

[@rajasdataengineering7585]

Thank you Kamal

[@gulsahtanay2341]

Great content, thank you!

[@rajasdataengineering7585]

Glad you liked it! My pleasure!

[@phanisrikrishna]

Hi Raja, that's a great lecture series on pyspark. Is there any possibility of getting the slides for reference?

[@alex45688]

I don't have such option as you have shown in community edition related to cluster, it is mentioned as your compute will automatically terminate after an idle period of one or two hours.

[@rajasdataengineering7585]

Databricks community edition has many limitations and this is one of that limitation

[@alex45688]

@@rajasdataengineering7585 ok can you share the data sample it would be helpful for practice

[@Kohli079]

Hi sir I don't understand use of spot instance?

[@SatishKumar-nr8ju]

Hi Raja Can you please provide Remaining Transformations also

[@rajasdataengineering7585]

Hi Satish, sure I can provide remaining transformations

[@bossofdata]

I am confused about the terminate after _ minute option what will happen after a specified time period is that cluster will terminate permanently or will that cluster restart again while executing the next query

[@rajasdataengineering7585]

It will permanently shut down after x minutes of idle time, if this option is enabled. So when we need to execute the next query, need to turn on the cluster manually

[@rajanib9057]

whats the difference between standard and single node what woukd be the driver /worker configuration for standard cluster

[@krisharjunakinjarapu3071]

Standard have both worker and driver single node will have only driver this driver only have to do execution

[@sureshkoduru8810]

Hi raja nice video, I have farced the one of the interview question what is cluster size in your project, how to configure cluster size

[@harishgangisetty7482]

Hi are you working related to data engineering

[@arupnaskar3818]

Hi Raja u r superb... 🌸 Thank you so much for your excellent work 💐💐.. Raja I am stuck I an situation... actually, I have a oltp postgres server and datalake as olap ... currently, all the customer service on my oltp server .... due to which we have indexes on oltp ... which is making huge latency on inserting ... COULD you pls suggest how to add a staging server .. which will get change feeds from my main oltp ... Please, help .. 🙏 thank u :)

[@rajasdataengineering7585]

Thanks Arup. If index is problem while inserting, you can think of partitin switch/exchange which is most commonly available in most of the databases. Also you can think of option where we can disable index while bulk loading and enable right after loading

[@arupnaskar3818]

@@rajasdataengineering7585 Thank you so much Raja .🍁.for replying ... Actuall, data coming as as streaming so bulk data load and applying index won't be a good option ...but I will do POC on partition switch.. thank u once again...

[@sankarazad7574]

Hi sir is there any python script to create the cluster automatically

[@rajasdataengineering7585]

Hi, yes it is possible through cluster APIs provided by databricks

[@sankarazad7574]

@@rajasdataengineering7585 can you please provide any link for reference here.