23. Databricks | Spark | Cache vs Persist | Interview Question | Performance Tuning

[@omprakashreddy4230]

Only few people have ability to teach in way that even novice can understand. Hats off to you. Keep going !!!

[@rajasdataengineering7585]

Thank you for your encouraging words

[@Poori1810]

can not agree more

[@gulsahtanay2341]

Thank you for sharing your knowledge with us!

[@rajasdataengineering7585]

My pleasure! Thank you

[@stepup2me1]

You have very good way of explaining the concepts. Thank you!

[@rajasdataengineering7585]

Thank you Chetan

[@pavithraeshwar8881]

This is the explanation thank you for share the knowledge sir👏

[@rajasdataengineering7585]

Thanks and welcome

[@joyo2122]

your videos are the best

[@tanushreenagar3116]

Nice content sir

[@rajasdataengineering7585]

Thanks!

[@rockykefunday2707]

you are the real raja bro , super

[@rajasdataengineering7585]

Thank you bro

[@kamalbhallachd]

Good 👍

[@rajasdataengineering7585]

Thank you! Cheers!

[@abinaya7704]

Your videos are making wonders!!

[@rajasdataengineering7585]

Thank you

[@rahulpandit9082]

I found many videos on KZbin regarding Cache and Persist, but nobody explain like the way you did...

[@rajasdataengineering7585]

Thank you Rahul

[@turanfair9364]

Best teacher!!! Thank you sir 🙏🏻

[@rajasdataengineering7585]

Thank you Turan

[@vutv5742]

Great explaination 🎉

[@rajasdataengineering7585]

Glad it was helpful! Keep watching

[@vlogsofsiriii]

Hi Raja. I have one doubt. Cache - will store the data in memory means is it onheap memory ?? Persist - Will store the data in onheap and off heap both ?? Is it correct ??

[@rajasdataengineering7585]

Yes that's correct. Cache always stores in memory but persist has flexibility of memory or disk

[@vlogsofsiriii]

@@rajasdataengineering7585 memory means here onheap rgt and disk means offheap??

[@rajasdataengineering7585]

No onheap and offheap both are memory and disk is different. I have already posted a video on onheap vs offheap. Pls watch that video

[@vlogsofsiriii]

@@rajasdataengineering7585 thank you 😊

[@ranjithajit4717]

Can you add the examples for creating persist in the description?

[@justvenkyy...3423]

this is too good . please keep doing. can you post on processing small file problem with spark?

[@rajasdataengineering7585]

Thanks 👍🏻 Sure will post a video for small file problem

[@shakthimaan007]

But where and how do we define these? Can you please add a short demo?

[@kamalbhallachd]

Knowledge session

[@rajasdataengineering7585]

Thanks Kamal

[@coolraviraj24]

You explained it so simply... i hope will be able to explain to the interviewer the same way u did😅

[@rajasdataengineering7585]

Thank you! All the best!

[@sanjayr3597]

Very good playlist which I have come across.. Could you please provide example with practical example because I was watching some videos regarding this and what I noticed was when we df.cache() then by default it is MEMORY_AND_DISK SER ..there was no just MEMORY_AND_DISK it was always SERIALIZED ..need to know the reason on this.

[@iamkiri_]

Raja, I really appreciate your explanation :)

[@rajasdataengineering7585]

Glad to hear that! Thanks for your comment

[@sravanthiyethapu9970]

Hi Raja, u said that persist will use both memory and disk. Here memory means both on and off heap memory????

[@rajasdataengineering7585]

By default, it is cached at on-heap memory. But if off-heap memory is enabled and jvm memory(on-heap) is full, off-heap memory would be used for caching remaining partitions

[@pankajchikhalwale8769]

I guess you have at least an M.Tech. + M.Ed. degrees. Expert in Spark and Amazing Teacher. Sir, Tussi Grett Ho !

[@rajasdataengineering7585]

Thank you Pankaj! Hope you like the tutorial

[@pankajchikhalwale8769]

@@rajasdataengineering7585, So far I have watched 9 out of the 22 videos in the "Databricks Performance Optimization" playlist. It is very detailed. Like it.

[@rajasdataengineering7585]

Glad you like it!

[@aayushdesai532]

great video sir! one question - is disc memory same as off heap memory?

[@rajasdataengineering7585]

No, off heap and in disc both are different. Off heap memory is part of RAM. on heap is controlled by jvm while off heap is controlled by os itself

[@RamaiahChenna]

Hi Sir, we want vidoe for performance issues and solutions while develope the notebook what are the issue comes

[@premsaikarampudi3944]

Hi, I was asked to prepare for Spark for my next role in the same company I am working, Is this learning series enough ?

[@rajasdataengineering7585]

Hi, yes this is more than enough if you complete all these videos

[@swathi6472]

Please make Video on Salting in Performance optimization

[@rajasdataengineering7585]

Sure will create a video on salting technique

[@suresh.suthar.24]

Best Explanation. but i have 1 question like cache() is a transformation or action ?

[@rajasdataengineering7585]

Cache is an action

[@tunyestark2633]

@@rajasdataengineering7585 No, cache is not an action.It is an transformation, please do try it out.

[@Uda_dunga]

Try to make videos under 10 mins sir

[@rajasdataengineering7585]

Sure, will do

[@MrPerikala]

how to avoid the duplicate rows while joining large datasets

[@rajasdataengineering7585]

Drop_duplicates or distinct can be used to remove duplicates