102. Databricks | Pyspark |Performance Optimization: Spark/Databricks Interview Question Series - II

Azure Databricks Learning: Performance Optimization: Spark/Databricks Interview Question Series - II
=================================================================================
Are you learning Spark/Databricks to become Bigdata Engineer? Are you preparing for an Interview for the role of Spark/Databricks data engineer?
Performance optimization is one of the constant topic in all interview calls.
Follow this video to get list of questions on performance optimization concepts along with directions to give answer in an interview
#SparkInterviewQuestions, #DatabricksInterviewQuestion,#SparkInterviewSeries, #DatabricksInterviewSeries,#SparkPerformanceInterviewQuestions, #SparkOptimizationInterviewQuestions, #DatabricksInternals #DatabricksPyspark,#PysparkTips, #DatabricksRealtime, #PysparkPerformanceOptimization, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Databricksforbeginners

[@vadderamu5422]

Thank so much sir, Very understandable explanation way.

[@rajasdataengineering7585]

You are most welcome

[@ashswinsubbiah3752]

You are a gem sir.

[@rajasdataengineering7585]

Thank you, Ashswin!

[@dvsrikanth22]

Nice video thanks for sharing👍👍, Sir can u please do a video on incremental load from adls to sql table using csv files and also maintain the audit log table for files when the file loaded and time and no.of records in that particular file. So that in next load notebook should pick up only new file from adls path.

[@rajasdataengineering7585]

Sure Srikanth, will create a video on this requirement

[@sabesanj5509]

Thanks for the part 2 video raja bro👍. Those topics has minimal points for all the questions but some unique and important points to be told in interview..

[@rajasdataengineering7585]

Glad you liked it!

[@sabesanj5509]

@@rajasdataengineering7585 yes bro very informative🙌🏽

[@rajasdataengineering7585]

Thanks!

[@Learn2Share786]

Thanks, can you pls make a separate video on question no.21 i.e troubleshooting OOM issue in real time?

[@rajasdataengineering7585]

Sure Fahad, will make separate video on this requirement

[@tallaravikumar4560]

How to tune a spark job which is already running in Databricks ?

[@rajasdataengineering7585]

You can enable Adaptive Query Execution to handle few performance bottleneck automatically. Also you can use spark UI to get few metrics about hour job and tune based on that information

[@kanikevenki9988]

Thanks sir for sharing this

[@rajasdataengineering7585]

You are welcome!

[@kanikevenki9988]

@@rajasdataengineering7585 sir I'm preparing for interview for spark with Azure. Can you make about Azure also

[@rajasdataengineering7585]

Sure Venki, will create videos on other azure services like ADF, Synapse, ADLS etc

[@kanikevenki9988]

@@rajasdataengineering7585 thanks sir for your valuable time and replying name .. I hope the calses will starts soon

[@rajasdataengineering7585]

Sure!