cache and persist in spark | Lec-20

[@gauravpp5768]

bhai itna detail aur clear me to kahi bhi nahi hai, appreaciate brother

[@dayanandab.n3814]

You literelly showed us CACHE and persist. Thank you Manish Bhai. Keep Going high..

[@dayanandab.n3814]

Smile on your face and genuine explanation really makes me immersed in the training. Thank you Manish bhai - Lots of love

[@praveenprakash9756]

You are pride to NIT family , a perfect mentor for DE . Thanks for creating such informative content and delivering it so naturally . You are such an inspiration !

[@OmprakashSingh-sv5zi]

Bhut badhiya super lecture Manish bhai

[@shubhamwaingade4144]

Awesome Explanation brother!!!! Going watch the entire playlist now! Please keep posting such amazing content...

[@aditya9c]

what a superb vedio. Keep making these vdos for us.....huge thanks

[@payalbhatia6927]

where could we see high cpu usage in spark UI when data coming from disk to memory and gets deserialized ?

[@harshitasija5801]

Really great Explanation

[@apoorvkansal9266]

Notes whatever Manish Bhai explained regarding caching of Dataframe(s) : Also, as we know that Dataframe objects get stored in Executor Memory Pool during computations i.e. transforms but this memory is short-lived i.e. only till the time a transformation is done on a Dataframe. So what we can do is that if our usecase is such that joins are happening on that Dataframe or on particular Dataframes and we need to : 1. store the required partition of the Dataframe or 2. save execution time, 3. re-calculation i.e. efforts(resources) of executor(s) 4. and the cost . Then we can use cache() so that required partition(s) of Dataframe(s) get stored in Storage Memory Pool. We already know that Memory Storage Pool is used for storing intermediate state of the tasks especially during Joins and also used to store cached data and Memory Eviction is done using LRU method.

[@AIBard-pk5bs]

Hi Manish. Thanks for your video. Its very good. Could you please clear one doubt of mine. At 20:43 Why is that there are 4 partitions created even when the size of dataframe is almost around 11MB which is smaller than 128 MB (default partition size). I don't see anywhere in the video spark.sql.files.maxPartitionBytes getting updated? Then what is the reason for above behaviour?

[@manish_kumar_1]

Default parallelism is applied to 4. I think because of that 4 partition would have shown there. Anyway I will check that

[@AIBard-pk5bs]

@@manish_kumar_1 Thanks for your reply

[@Speed-0-meter]

Please make video on 1. dataset vs dataframe vs rdd 2. Hive with spark 3. Checkpointing

[@dataplumberswithajay]

great video bhaiya

[@nanochip1908]

can u pls make a video on serialization and deserialization.

[@tanushreenagar3116]

Best content sir 🎉

[@algorhythm3103]

Nice video sir!

[@Ks-yi8ky]

Supperb sir..

[@upendrareddy5880]

Hi manish, Which one is correct? For df. Show() it gets one complete partition, in that only 20 rows are to be taken by default for df. Show() it gets default 20 rows from across all the partitions(not from single partition)

[@TechDude3310]

Option 1

[@luvvkatara1466]

Thank you Manish bhai for this lecture

[@Vinodkumar_19879]

How to connect for you

[@raajnghani]

How can we unbroadcast any dataframe?

[@Matrix_Mayhem]

Thanks Manish!

[@subrahmanyanmn9189]

Manish Bhai, Thanks a lot for the video series. Its really helpful. 1 doubt, since Spark removes data stored in memory on basis of LRU, then is it compulsory/necessary to unpersist?

[@ritikpatil4077]

Good question, but the only usecase i could think of this. If you have big dataframe (still need to be smaller than storage memory 😁) and u using it repeatedly. U can cache it. and after your usage u can unpersist it so u save storage memory.

[@amankumarsrivastava6852]

Bhai ye saare pyspark dataframe api ke functions kahan se practice karu , bhulte jarha hoon , koi site ya practive platform ho jaha practice kar saku plz suggest .

[@mallangivinaykumar9500]

Your way of teaching is awesome. Please make videos in the English language.

[@da_nalyst]

Manish bhai thank you for great explanation. Ek doubt h. Df jo cache kiya, vo sabhi executor pe alag alag store hota h ya kisi bhi ek executor ki memory me store hota.

[@nanochip1908]

Hi Manish, How much Jio pays to 3 YOE in data engineering ?

[@arpittrivedi6636]

Sir app big data scratch se sikhte ho kya? Please share video id possible

[@pratyushkumar8567]

Manish bhaiya thoda specfic please explain as we know cache data storage memory pool mai present hota hai So when we do .cache() on the dataframe the result of it stored in storage memory pool or executor memory pool? please explain

[@prashantmehta2832]

same question.

[@mmohammedsadiq2483]

Can you provide spark documentations URL and reference books , Thanks

[@aneksingh4496]

Very very informative .. Can you please make videos on spark scenario based

[@piyushjain5852]

what do you mean by recalculating the partition when it exceeds the memory?

[@manish_kumar_1]

Jo memory store nhi hua usko DAG ke help se recalculate karna parta hai if needed

[@piyushjain5852]

@@manish_kumar_1 lekin recalculate krne k bad bhi store to krna hi padega na fir vo kse handle hoga jb phle ni ho paya?

[@manish_kumar_1]

@@piyushjain5852 wo store nhi karega. Jitni baar v aap same df ka data use kariyega utni baar data recalculate hoga. Memory to storage memory pool ka full hai naa ki executor memory pool ka. Isliye hi to performance impact Hota hai.

[@Daily_Code_Challenge]

thank you

[@madanmohan6487]

Please make video on dimension table vs fact table

[@ajinkyadeshmukh2343]

manish bhai cache() default storage level MEMORY-ONLY hai aap please ik baar spark Documnetation check kar lena

[@sanooosai]

thank you sir

[@pramod3469]

Thanks Manish very well explained when we choose memory only you said if partition size > memory available then it will re calculate What will be recalculated here ?

[@syedadnan4910]

the partition that is not stored in the memory will be recalculated ,as we know that it follows full partition principle only

[@amanpirjade9]

Make video on SQL for data engineering

[@manish_kumar_1]

Sure

[@adityakvs3529]

Bhai, in dag there are no loops right then how would it recalculate

[@shobhitsharma2137]

sir i want to talk to you in person

[@manish_kumar_1]

Link description me hoga

[@pawansalwe1926]

tq

[@Speed-0-meter]

Spark streaming no one is talking. Would be good if u throw some light.

[@Vinodkumar_19879]

Manish bhaiya aap se baat karna hài

[@raghavendrakulkarni3920]

Itna time kyun ? Bahot jyada gap why?

[@manish_kumar_1]

Haan Thora busy tha

[@raghavendrakulkarni3920]

@@manish_kumar_1 accumulator pr kab laoge video And salting on skwed data

[@Vinodkumar_19879]

Aap meri help karo

[@Vinodkumar_19879]

Please please

[@sankuM]

bhai @@manish_kumar_1, some issue with the video quality.. only 360p is coming up even thought it's been 7 mins of the video upload!! 🤨🤨

[@manish_kumar_1]

May be 4k process nahi hua hoga tab tak