Spark Scenario Interview Question | Persistence Vs Broadcast

[@mrkrish501]

This scenario was not clear when i go through external video but after your explanation i understood the difference. excellent

[@rameshthamizhselvan2458]

Excellent... Billion-dollar video...

[@sumitgandhi628]

Thanks for explaining it in easy way :)

[@srinubathina7191]

Super content thank you

[@umamahesh2307]

Data in memory and data in disk will not occupy same space. So if it is 12gb on disk in memory it can be 18gb

[@gemini_537]

Is it approximately 12GB after persisting? Any significant overhead when the data is in memory?

[@Jayaditya87]

if we persist with serialize, we will have cpu overhead else no overhead.

[@HemanthKumar-cm9lv]

Hi viresh, Thanks for the video, can you confirm the below statement, In Persist executors save partition of the data frame in memory and in broadcast executor will save the entire data frame in memory?

[@mateen161]

Hi Viresh, Thanks for this nice video....I believe that the broadcast variable is used to broadcast a small table and join it with a huge table which would avoid shuffling....What happens if we broadcast a table with more number of columns into the executors? Assuming that the broadcast table is larger in size because of having more number of columns.

[@dipanjansaha6824]

You would not be able to leverage the benifit of broadcast join in that case. Even though you forcefully enable broadcast, would not be noticing much impact or improvement

[@mateen161]

@@dipanjansaha6824 Sounds good...Thanks

[@TechWithViresh]

This would fill up the memory quickly and can result in out of memory issue , that’s why default size for auto broadcast is 10 mb, Also as discussed in separate video, memory footprint in broadcast becomes 4 times of the actual size. Thanks

[@mateen161]

Thanks Viresh for the information.

[@hiteshpatil5906]

why here u r taking only 3 no. of executor?

[@mahendramaurya3208]

why the number of partitions are three in case of broadcast join, Can we keep it low like 1or 2 orr why not to keep all the partitions into single executor

[@TechWithViresh]

It will throw out of memory in that case for sure.

[@gemini_537]

In the case of broadcast, why do we have to include the 12GB of existing DF? I feel it is unfair to compare persist with broadcast. It is possible to avoid the 12GB?

[@Jayaditya87]

it is the primary /parent data using which we have created copies and shared it to executors. until this application ends this parent data is still part of the program.

[@siddhantpathak6289]

Hi Viresh, I am new to Spark so cut me some slack for asking newbie questions. In persistence, the data frame is held either in memory or disc. Suppose the data from the data lake was held in executor memory which is in the given case is 4 GB and it is completely occupied. Now I want to read another data frame, then how executor will deal with it since its memory is already occupied by the previous persisted data frame. In Broadcast, the memory footprint is said to be 4 times the data frame memory. Where does this memory come from since each executor got only 4 GB? And I also read somewhere after Garbage Collection it is only left 3 times. Why is it so? In persistence, the data is said to be stored in memory. Is it just the executor memory which is 4 GB or entire system memory? Thanks in Advance.

[@suryasatish5581]

broadcast variable is copy per node right ? why it will 36 GB ?

[@MrVivekc]

what I understood in case of broadcast variable, 12GB data is copied to all, 12 GB per node, nodes thus adding upto 36GB.

[@aneksingh4496]

Hey Viresh , from where do you study these concepts . please share resources

[@anantababa]

not clear ..you have explain about the persistence ..

[@ITEraTech]

no voice clarity

[@vkd9442]

Not clearly explained

[@TechWithViresh]

Please post your doubt/question, we will try to answer the same. Thanks for the feedback.

[@skms31]

@Technical Tutorials : Please be specific in giving feed back so people can help you !Be as specific as possible please