Deep Dive: Apache Spark Memory Management

[@oldoctopus393]

the answers to almost all my questions on the topic are given in 26 min video. Awesome!

[@sankarramanathan6051]

This was a really good presentation. Very unassuming, to the point with lucid slides.

[@laeeqahmed1980]

It would be nice to have this kind of execution, storage visualization in Spark UI.

[@harshitsaini15]

Its available at executors/storage tab, port 4040 by default.

[@ankireddyambati7638]

Great Talk

[@anfield6321]

kpii left TNC for this? NotLikeThis

[@joo02]

my head = 1min.:D

[@kk-si6fy]

Since memory can always spill to disk when do we ever run out of Memory?

[@verma.chitral]

kk8866 you run out of memory when you try to accumulate results that have a size greater than the memory you have on driver as a result of some action like take(). it also happens when you are making large objects.

[@vikashpareek8374]

@@verma.chitral This is the case when the driver runs out of memory. But why executors run out of memory?

[@vikashpareek8374]

Did you get the answer to why executors run out of memory when memory can be spill to disk?

[@verma.chitral]

Easy, it happens due to gc overheads and oom s. When you're creating objects at greater rate they can be gc'd.