Spark Monitoring: Basics

[@cchalc-db]

How am i the first person to like that video? That was one of the best and most concise explanations I have seen. Thanks!

[@DustinVannoy]

Thanks, glad you enjoyed it!

[@kamalkunjapur5383]

Great video!! Much appreciate the effort put in to add join section, Dustin.

[@manasr3969]

Amazing content , thanks man. I'm learning a lot

[@thevision-y1b]

is the spill memory bad? @3:48

[@DustinVannoy]

@@thevision-y1b yes, it’s not ideal. Indicates I either want to 1) change to a worker VM type with more memory per core or 2) split into more tasks since the Input Size for my median and max tasks is a bit too high. By the way, these days that input size is usually ok for me but I use different node types now.

[@mkhannautube]

It would be nice if you could someday explain how to ship the logs outside of Databricks and into systems like Azure Log Analytics Workspace or ElasticSearch. Great video by the way

[@DustinVannoy]

You can do a lot by setting init scripts but I haven't done it to forward to ElasticSearch. For Log Analytics there is a library (currently only for DBR < 11). I cover that here: kzbin.info/www/bejne/nJzXq2lpqqmtg5Y

[@film-masti-777]

@@DustinVannoy Hello Dustin, so you mean for DBR>11, this method wont work? Any suggestion on alternatives we can use for DBR >=12 to bring Log4J output and databricks cluster event logs into log analytics workspace?

[@jwc7663]

Does kzbin.info/www/bejne/iZ6caa19m6aArKM only for Databricks? I can't see this tab in Spark UI. Neither does heap histogram

[@DustinVannoy]

Correct. Though other environments may include it, it isn't part of open source Spark. It would need to be setup separately.

[@jesica9540]

So am I the second comment?!