04. On-Heap vs Off-Heap| Databricks | Spark | Interview Question | Performance Tuning

#On-HeapMemory, #Off-HeapMemory, #On-heapVSOff-heap, #SparkMemoryManagement #DatabricksOptimization, #SparkOptimization, #DatabricksInterviewQuestions, #SparkInterviewQuestions, #DatabricksInterview, #DatabricksPerformance,
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@datawallah1289]

Thank You for such a wonderful explanation. Really helps a large community of people persuing in big data

[@rajasdataengineering7585]

Thank you Atul

[@abhinavsingh1173]

@@rajasdataengineering7585 Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks

[@epicure07]

Very good video, thank you

[@rajasdataengineering7585]

Glad you liked it! Welcome!

[@nsrchndshkh]

Beautifully explained, keep up the good work. Waiting for your next video.

[@rajasdataengineering7585]

Thank you

[@akashhudge5735]

why no one is telling how to remove the data from off-heap memory? what happens when off-heap memory is full as there is no automatic garbage collection? but explanation is good.

[@RadomName3457]

very good tutorial so far, thanks sir!

[@rajasdataengineering7585]

Glad it was helpful!

[@sivasankar2968]

Very help full

[@rajasdataengineering7585]

Thank you

[@saravninja]

Good explanation!!

[@rajasdataengineering7585]

Thank you Sarav!

[@vishalaaa1]

excellent

[@rajasdataengineering7585]

Thank you! Cheers!

[@nic-iv4ds]

Great content, can you please show some example of using huge dataset without off-heap memory and with off-heap memory

[@rajasdataengineering7585]

Sure Niharika, will make an example and post a video

[@ritchie6655]

There is Garbage Collector to clear the memory in on heap Memory, what happens when the Off-heap memory becomes full?

[@Wiztech360]

Great content, I was wondering if you can provides the notes or the ppt's of the tutorials.

[@LearningBigdata]

Thank you for the awesome series. The videos are missing in the series. After 4th video, the videos are missing and out of 100+ video series, only 65 seems to be available and videos are missing in-between. Could you please suggest?

[@rajasdataengineering7585]

All videos are available but they are segregated based on a playlist. If you look at videos under "videos" tab instead of playlist, you can find 113 videos overall

[@LearningBigdata]

Thank you so much for the reply. May I ask for your LinkedIn or any other way to get in touch with you, please?@@rajasdataengineering7585

[@gulsahtanay2341]

Very helpful

[@rajasdataengineering7585]

Glad you think so!

[@pavithrakumaraguru8487]

It would be great if you could share the notebooks and PPTS used on videos.

[@user-kq9cm1uf8g]

Great! can you please tell how to handle the out of memory exception in spark

[@suriyams3519]

HI raja, 1. when we enable off-heap memory gc will not come in to this picture ?- 2.) gc memory will be used when off heap memory is not enabled? 3.) off-heap used to store the intermediate results instead of disc?

[@rajasdataengineering7585]

Hi Suriya, yes these understanding is correct

[@suriyams3519]

@@rajasdataengineering7585 Thanks:)

[@suresh.suthar.24]

Thank for video, but i have small doubt like when we use persist() method with storage level off-heap then data will be stored in techyon or worker node's off-heap memory ?

[@rajasdataengineering7585]

Off heap memory of Worker nodes

[@suresh.suthar.24]

@@rajasdataengineering7585 thank you 🙏🙏

[@YoSoyWerlix]

Hi Raja! Great content, shall we follow the playlist as it is or by following the numbers before the title? (Asking for the best order for a beginner) thank you!

[@rajasdataengineering7585]

Hi, thanks for your comment. You can follow the number before the title

[@dineshdeshpande6197]

@@rajasdataengineering7585 , The video sequence is missing for the numbers in title, makes its quite confusing for the order to follow. The playlist has 65 videos but numbers in title are 100+ . PL help here to follow the proper order.

[@sophiayue2484]

Well Done! Is it possible you might share the slide with us? Thanks.

[@suriyams3519]

off heap memory and memory overhead are same?

[@rajasdataengineering7585]

No both are different

[@suriyams3519]

@@rajasdataengineering7585 Thanks::::)

[@shivanireddy1428]

@@rajasdataengineering7585 can you also share what is overhead memory and how is it included in executor memory

[@dq5507]

Hey subtitles will be more helpful. PLEASE ADD IT.

[@satishkoppisetty927]

please do video on skew data

[@rajasdataengineering7585]

Sure Satish, will post videos on skew data

[@rajasdataengineering7585]

Hi Satish, as per your request, posted a video on data skew topic. Hope it is useful to you kzbin.info/www/bejne/e4LLnZevgbyDras