Broadcast vs Accumulator Variable - Broadcast Join & Counters - Apache Spark Tutorial For Beginners

This video session will explain what are braodcast variables and accumulator variable in spark and covers the following topics-
What are broadcast variables in spark-
How can you broadcast small dataset to worker nodes in spark?
What are broadcast joins in spark?
What are accumulator in spark?
How to create counter variables in spark?
Broadcast vs accumulator variables?
How spark jobs work internally?
Real exapmple on the usage of broadcast and accumulator in spark
Caching in spark
Write only variables in spark
Limeguru Website:
www.limeguru.com
LimeGuru KZbin Channel
/ limeguru
Limeguru Facebook Page
/ limeguru

[@madhu1987ful]

The best explanation so far I found on KZbin...easily explained

[@Rafian1924]

You are the best trainer on KZbin bro. Keep up the good work.

[@abhishekfulzele3148]

In addition to the Resilient Distributed Dataset (RDD) interface, the second kind of low-level API in Spark is two types of “distributed shared variables”: broadcast variables and accumulators. These are variables you can use in your user-defined functions (e.g., in a map function on an RDD or a DataFrame) that have special properties when running on a cluster. Specifically, accumulators let you add together data from all the tasks into a shared result (e.g., to implement a counter so you can see how many of your job’s input records failed to parse), while broadcast variables let you save a large value on all the worker nodes and reuse it across many Spark actions without re-sending it to the cluster.

[@anujasebastian8034]

I've been looking so many videos...It is only now i got the concept...thanks so much for the explanation.

[@ashutoshranghar2952]

Bro best Explanation WOW>>!!!.Also, do you have a video of explaining entire SPARK-SUBMIT command as to how the worker nodes are created and data is distributed across multiple partitions and task and jobs?It would be really helpful

[@learnwithfunandenjoy3143]

Excellent explanation... Great video to learn the concept in so a simple way. Please make another video so that we could learn all such concepts easily. Thanks.

[@afaque67]

Hi, Many people have questions how accumulator is getting update. Accumulator variable on each worker node is a local copy and there is a global copy which is in driver node and it can be accessed only by the driver process... Hence each worker node will return the count of blank lines to the driver process and the driver process will cumulate and update the global copy.

[@svcc7773]

Exactly

[@architsoni3669]

Yes true, this explanation is half cooked

[@svcc7773]

It's clear and nice explanation. this is one of best vedio so far in this concept thanks

[@rajeshguddati210]

Thank you sir, with simple example

[@ca20215]

Excellent explaination.

[@arunasingh8617]

It's informative, Can you also let us know in what situations accumulators is useful?

[@Shubhaarti2501]

Excellent Teaching

[@VivekKBangaru]

clear explanation thanks buddy

[@kurakularajesh4617]

super bayya, nice explanation

[@rajatsaha891]

Awsome explanation

[@kishorekumar2769]

excellent video bro.Great explanation and very thorough

[@drdee94]

Excellent explanation!

[@mangeshpatil714]

Nice explain sir.. 👌👌👍👍

[@atheerabdullatif7557]

amazing!

[@prabuchandrasekar3437]

Thanks for the clear explanation

[@soutammandal8839]

Bro u r champ nice explaning

[@adarshnigam75]

Awsome explanation..!!

[@bharathkumar-eg3gc]

You said that accumulator value is being updated in each worker node, does worker node 2 will wait until worker node 1 empty lines count updated done? since you are updating the value........... AS SPARK JOB IS A PARALLEL HOW COULD IT GET UPDATED SEQUENTIALLY?

[@hiItsEshikahere]

i have the same question as well

[@airesearch8057]

@@hiItsEshikahere I think each worker will have its own version of the accumulator (local accumulator), and each worker will update the state of its own local accumulator and when the workers finish the processing, the local accumulators will be sent back to the driver, and the driver will aggregate them all into the global accumulator.

[@harshadborkar2550]

​@@airesearch8057This is the correct answer, workers will have their local variables cached once work is done it sends back the results to the driver node and gets merged.

[@shreyash18]

Time stamp 3.55 spark submit .... You didn't mentioned about cluster manager role in spark submit background process As u mentioned drive program initiate and connect to worker ....yet driver connect with cluster manager and cluster manager wil connect to workers

[@merimihelmi8626]

thank's for this explanation

[@bhavaniv1721]

Thanks for sharing such a nice video can please share me spark scala training videos

[@BetterLifePhilosophies]

Yes Thank you.. my questions is how the situation will be handled in case we have encountered blank lines at same time on three worker nodes?

[@mayankvijay3436]

I don't think in broadcast variable example what you showed that w1 contains only USA and w2 only IND is correct. Data is distributed in random fashion and code map can be used as lookup within that worker. Please correct if understanding is wrong.

[@chetan30081991]

I think since broadcast variable is of small size, it will share the complete code map over all workers without segregating the data

[@dhananjayreddy9998]

When the data is getting analyzed parallelly, then how come the Accumulators get incremented. For example partition 1 has 1 space line and partition 2 has one space line, when these two processed simultaneously, both partitions can update the accumulator as 1 right. Could you please clarify

[@kashishshah8417]

can i have the accumulator variable pass the value to broadcast variable? Like some worker nodes update the accumulator variable which is copied to a broadcast variable and inturn read by some other worker nodes

[@haveafuninlife]

broadcast variable is immutable. once you do broadcast from driver node, value of the variable is sent to all the worker nodes. Workers can just read the value.

[@svcc7773]

Didn't mention how to retrieve record from broadcast variable

[@architsoni3669]

This is not the correct explanation for Accumulator variables from the start. Kindly edit the video to add factual information

[@shikhersingh5026]

This guy said, driver will create worker node. I think he should review his video before posting. Every single person is just want to make money by starting his own channel but does not want to spend time in giving quality videos.

[@bollytv8305]

So many ads