Spark Architecture in 3 minutes| Spark components | How spark works

Spark is one of the most prominent and widely used processing framework in Bigdata world. This videos explains the core components and architecture of spark with a real world example in just 3 minutes.

[@satishchippa]

Excellent way of explaining things in a most simplified manner. Looking forward to more videos on Spark.

[@BigDataThoughts]

Thanks Satish

[@deepalirathod4929]

Finally it got cleared to me after reading here and there . thank you .

[@bitthal24]

I havent seen such a lucid way of explaining something this complex concept. Great work!

[@BigDataThoughts]

Thanks Bitthal

[@showbhik9700]

This is the only video on KZbin which clarified my doubts. Thanks!!

[@BigDataThoughts]

Thanks showbhik

[@BigDataThoughts]

Thanks showbhik

[@nareshkumarbattula5847]

It's the best video I've seen so far on spark architecture..awesome..keep going..

[@BigDataThoughts]

Thanks

[@nahomg.4191]

I wish I could give 1000 likes. You’re an excellent teacher!

[@BigDataThoughts]

Thanks

[@theamithsingh]

finally, a video that simplifies spark, amazing, keep the videos coming please!!

[@BigDataThoughts]

Thanks

[@sandeepchoudhary4900]

Very nice video and it covers everything related to the Spark architecture in just 5 minutes. Keep sharing new videos.

[@BigDataThoughts]

Thanks sandeep

[@mohnishverma87]

Just woow, very simple explanation of a complex cluster overview.. Thanks.

[@BigDataThoughts]

Thanks

[@rahulkoley9447]

Very Informative, with full of clarity, Thank you.

[@BigDataThoughts]

thanks

[@apurvgolatgaonkar6722]

Ma'am you taught amazing 😍😍 very less time consuming lecture but perfect... Keep it up

[@puneetojha1195]

This video is better than 1 hour course on spark . Thanks

[@BigDataThoughts]

Thanks puneet

[@vsriga82]

Lot of information presented in simple way for everyone to understand. 👍

[@BigDataThoughts]

Thanks sriganesh

[@desiengineerashish4908]

This made far easy to understand Spark Architecture. Thank u ma'am, you are great

[@BigDataThoughts]

Thanks Ashish

[@seemanthinin448]

Simple example and easy way of explaining an important concept.. thanks!

[@BigDataThoughts]

Thanks seemanthini

[@krishnavardhandasari2694]

Thank you mam for excellent way of teaching Spark.

[@BigDataThoughts]

Thanks

[@Azureandfabricmastery]

Simple and easy to understand. Thanks. I like doodle way of explaining concepts :)

[@BigDataThoughts]

Thanks sheik

[@rovashri566]

How did you make such a good visual explanation? Which tool you used to draw sketches ? Pls guide 🙏

[@adityasisodiya3000]

Amazing content ! Really appreciate the understanding and approach to explain...looking fwd to more

[@BigDataThoughts]

Thanks Aditya

[@dancingmoveswithdhruv3649]

Very clearly explained, really appreciate all your efforts

[@BigDataThoughts]

Thanks Dhruv

[@iamramanr4s]

way of explanation is .....just amazing

[@BigDataThoughts]

Thanks

[@NIYANTAjmp]

very simple and nice explanation. Thank you for posting this video

[@nishchaysharma5904]

Thank you for this video.

[@BigDataThoughts]

Thanks

[@ankitachauhan6084]

the best explanation ever great work !

[@BigDataThoughts]

Thanks

[@minaksheebagul4938]

just love the way u explained it ;) Really appreciated mam.

[@BigDataThoughts]

Thanks minakshee

[@mehrozalam94]

Great Video, I have learned alot. Thank you

[@ayyappareddymuthikepalli4261]

Nice explanation.. Pls keep videos 🎥 like this

[@SagarSingh-ie8tx]

Example was very good for beginners

[@neerajmishra6828]

Saw this video.. content looks promising... great job

[@BigDataThoughts]

Thanks

[@vikastangudu712]

one of the bests

[@BigDataThoughts]

Thanks vikas

[@lakshmikanthavilalakumar]

Beautifully explained short video 👏👏

[@BigDataThoughts]

thanks lakshmikanth

[@tanushreenagar3116]

VERY HELPFUL BEST EXPLANATION

[@BigDataThoughts]

Thanks

[@athar5867]

Your videos really help me in clearing my interviews and getting the job, now I need some support for my new job. Can you either post videos regarding how pyspark class object work in backend, how parquet/csv reading writing work in distributed environment like which data read by each executor, how to do pagination of some order data etc. If possible provide your LinkedIn profile url or suggest some way to connect

[@SaurabhKumar-mc1is]

Vividly explained. Thanks mam

[@BigDataThoughts]

Thanks saurabh

[@ThankGod143]

Extraordinary mam

[@BigDataThoughts]

Thanks harika

[@toandaominh1997]

This was marked to know that I'm here on 3/10/2023

[@trainingt9855]

Great video

[@vemulasunayana904]

Excellent example 👏

[@BigDataThoughts]

Thanks

[@vedantshirodkar]

Mam, I have one question. If spark has to write a data to sql database and as our data is broken on to multiple worker nodes, so is it driver who establishes single connection with sql db or it is worker nodes who establishes multiple parallel connections ?

[@BigDataThoughts]

When Spark writes data to a SQL database, it is the driver program that establishes a connection with the database and manages the write process. Each worker node will write its portion of the data to the database through this single connection established by the driver

[@vedantshirodkar]

@@BigDataThoughts Thank You Mam for the elaborated explanation.

[@moughosh3640]

Extremely good explanation

[@BigDataThoughts]

Thanks mou

[@hlearningkids]

Very nice 👍

[@BigDataThoughts]

Thanks

[@hlearningkids]

@@BigDataThoughts did you explained in this style big query also. ? improvement in this video can be summary in slow way. please dont get hurt because i gave comment. you did really well in video. excellent explanation.

[@askdoubts6359]

Grate nice explanation

[@zub3rahmed76]

Nice mam

[@gkethanvarma889]

Thank you so.................. much for this

[@MrSmarthunky]

Really informative Shreya. One quick question. Stage will run sequentially depending on use case and Tasks will run in parallel?

[@BigDataThoughts]

Thanks Madhu. Stage may run sequentially or in parallel depending upon whether they have dependency or not. Typically a stage will have multiple tasks running in parallel on a different set of data but doing the same set of operations that the stage contains.

[@samk_jg]

Amazing!

[@sanjoydas007]

Very helpful, it would be great if you can take an example and illustrate how the data chunking happens

[@sheereenhamza3700]

Thank you for such a goooood explanation :D

[@BigDataThoughts]

thanks sheereen

[@upskillwithchetan]

Great explanation Ma'am, please add more videos and arrange it in seq. under playlist

[@BigDataThoughts]

Thanks Chetan. yes there are more videos coming up. stay tuned

[@nareshb5859]

Very Nice Explanation

[@BigDataThoughts]

Thanks Naresh

[@hemanthkumarreddyedde]

Superb delivery

[@BigDataThoughts]

Thanks Hemanth

[@vidyaradhakrishnan5611]

Very nice explanation,

[@BigDataThoughts]

Thanks Vidya

[@shreyamoghe6893]

Great video! I have one question though. Is it my correct understanding that each student which got the coin bag is same as how data is partitioned. i.e. 1 student = 1 data partition?

[@BigDataThoughts]

1 data partition is operated on by 1 slot/task

[@RaviYadav-cx2pb]

Amazing explanation mam 😊😊👍

[@BigDataThoughts]

Thanks ravi

[@peekagyan]

Nice explanation. For 1 GB input data for a batch processing how can we decide how can we decide the cluster size, no. of nodes or no. of executors ? Could you please explain Thanks ma’am

[@SandyRocker]

Thanks a lot mam

[@BigDataThoughts]

Thanks Sandy

[@SandyRocker]

Subscribed ✅

[@shivratanmishra1459]

Great work ma'am

[@BigDataThoughts]

Thanks Shivratan

[@mohanmannam]

good explanation...

[@BigDataThoughts]

Thanks

[@iwonazwierzynska4056]

Excelent video :)!

[@BigDataThoughts]

Thanks

[@Ramakrishna410]

One executor in one core and 2 partitions are assigned so one by one will execute. My que is if 10 tasks are there then these tasks wil execute parallelly or sequentially in partition level

[@BigDataThoughts]

A task operates on a partition of data. Tasks do run in parallel. If you have multiple cores you can specify how many cores will a executor use. The number of concurrent tasks an executor can run is equal to the cores assigned..

[@swapnilchilwant6867]

Thank you ma'am..👍

[@sandeepmullangi4413]

Nice video. Really liked it. So you said one node can act as driver. I want to know what is the best practise here? I usually submit jobs by doing SSH to master node (atleast in GCP dataproc) and then submit job. So should I consider my master node as driver? Is it right to do that way?

[@BigDataThoughts]

To give an example if we are using Yarn - when you are submitting a spark job in cluster mode. Container where the Application Master runs acts as Master node (driver) and the containers where all the executor process runs the tasks are called Slave Node. When the job gets submitted first the spark submit calls resource manager which in turn starts the application master and from there driver takes over.

[@ur8946]

could you pls explain more on parttiton

[@BigDataThoughts]

The Dataset is divided into partitions and each partition is the unit on which a task works. That's the input to task.

[@vibhad-cv4sf]

Great Video....!! Appreciate your efforts!🎉 One question, Where does a cluster manager fit in in this architecture? What role does it play in comparison with driver?

[@BigDataThoughts]

Cluster manager's job is to provide resources for job execution. Ex - yarn, mesos etc. Driver is the one controlling the overall job execution and which executors take part in the job

[@vibhad-cv4sf]

@@BigDataThoughts ohh okay!! Thank you!!

[@bintangmuhammad7082]

Can you please turn on the subtitles? thank you

[@RameshKumar-ng3nf]

At the start of the video i was so happy seing all the diagrams.. Later got fully confused & felt complicated and i didnt understand well 😢

[@SM-mq1iq]

Can you make it slow to follow. I felt this was fast to get to know the terms.

[@trainingt9855]

Can you help in understanding RDD

[@BigDataThoughts]

RDD are resilient distributed datasets and they are the lowest abstraction in spark. check this video - kzbin.info/www/bejne/eqa6eHWDe5x0iMU

[@mdatasoft1525]

❤

[@sumonmal009]

Good playlist for Spark kzbin.info/aero/PL1RS9FR9qIPEAtSWX3rKLVcRWoaBDqVBV

[@BigDataThoughts]

Thanks

[@harigovindk]

18/april/2024