Learn Apache Spark in 10 Minutes | Step by Step Guide

[@DarshilParmar]

Don't forget to hit that Subscribe Button for more amazing content :)

[@0to1_war]

Get ready with project.

[@ashishkumar-ns3sg]

Please also upload GCP data engineering End-to-End project

[@cs_soldier5292]

You deserve much more than 1000 buddy. I learn so much from your channel

[@BabaiChakraborty-ss8pt]

lets get that.

[@NirajAgrawal-e6v]

Are you from Gujarat?

[@AxelNtwari]

Bro cooked. From the history, to the technical design and demo! Hats off!

[@rajdeepnaha1242]

00:00 Big Data and Hadoop 01:25 Hadoop processed data in batches and was slower due to disk storage, Apache Spark solves these limitations. 02:43 Apache Spark is a fast and efficient data processing framework. 04:11 Apache Spark is a powerful tool for processing and analyzing Big Data. 05:42 Apache Spark application consists of a driver process and executor processes. 07:02 Spark data frames are distributed across multiple computers and require partitioning for parallel execution. 08:24 Spark transformation block will give the final output. 09:40 Spark allows the conversion of data frames and the execution of SQL queries on top of it.

[@jjones40]

Thanks for actually explaining spark, instead of making general comments or assuming we know the basics. Great video. Thumbs up, subscribed.

[@Remoes02]

agreed. I watched like 5 videos prior to this one that made wild assumptions about what I knew

[@thealbaniandude1997]

That was an extremely good explanation. Not only explained the theory but also practical examples.

[@krneki6954]

best explanation on spark in 10 minutes. its like feynman explaining physics. excellent job!

[@aparnapandian13]

I had understood the concept clearly within 10 min . Now I had a great understanding and knowledge about Apache Spark . This is best Spark video I had gone through . Its clear and top notch Explanation about each of the topics .

[@sureshlira6307]

I never knew I could recall so much in just under 10min... Wonderful content and well explained keeping it simple...

[@DarshilParmar]

Glad you liked it

[@fatihkeskin5867]

I was waiting for this. Please share an end to end project using Spark.

[@DarshilParmar]

Yes

[@Rider-jn6zh]

Waiting for the same...right from spark installation on local as well as on cloud platform

[@sumant542]

Please upload ASAP.

[@nirakarsahu4844]

Yes, if possible can you please also share using pyspark as well..

[@Rahul-fq9kf]

You are doing a fabulous job of making Data analytics so easy for everyone. Thank you so very much. God bless you!

[@devarapallivamsi7064]

I usually be off from content titled learn/master/excel X in Y minutes. would have definitely done the same had I came accross this by myself. Watched it only because my frd shared to me. Now I feel that I am lucky after watching this as I could wrap my head around SPARK. Subscribed.

[@kranthikumarnagothu3056]

Such a nice content! What a man you are! You have covered everything in spark in just 10 mins. I wonder how you made this video and the effort u put in to make this video is wonderful. Thank you for sharing nice content in such a simple manner!!

[@sajaljaiswal1659]

This was insanely good. Thanks for explaining the basics so clearly. Now I can learn deeper more comfortably.

[@jyotikothari499]

Apache Spark -- explained core concept in such a simple language.. Wonderful job 👍👍👍

[@prateeksachdeva1611]

The best Spark tutorial I have ever gone through. Thanks a lot Darshil.

[@DarshilParmar]

Wow, thanks!

[@mathewsignatious2838]

Excellent explanation-clear, concise, and straight to the point.

[@manyumungara1081]

I didn't understand apache spark since my undergraduate until I found this gem.

[@youhavetodoit9070]

Thanks For Explaining in 10 Min 🙌

[@tortingseloc1810]

The first very clear video about spark that I have seen.

[@shobogenzo]

As many other already said, tantastic and informative video on Spark. Nice context by providing the history of Hadoop. Nice pace too, not to fast, not too slow!

[@VanshSingla-jp4jy]

Alright, but need a full tutorial on this topic, if you can.

[@DarshilParmar]

Working on it!

[@anasshaikh3885]

​@@DarshilParmarthank you please upload it ASAP

[@ajinkyagaikwad4181]

​@@DarshilParmarplease upload

[@adieka1633]

​@@DarshilParmar this is a what the heroes did. Kudos to you Darshil

[@newbie8051]

Saw a bunch of your roadmap videos back in my freshman year, and now back here prepping for my DS internship, thanks ! The job description had spark/mapreduce which brings me here : )

[@newbie8051]

I tried to replicate the code block at 10:13, Can we use tips.filter(filterA & filterB), this applied both filters at the same time and does not create intermediate results tips.filter(filterA) will create some dataframe, which will be filtered by another filterB Please correct me if I'm wrong thanks !

[@njokiwambui3447]

Thanks for this.Currently reading spark definitive guide.Looking forward to full tutorial

[@DarshilParmar]

Coming soon!

[@xfarooqi]

So in just 10 mins, I get to know about Big Data, Hadoop, Spark, Pyspark and how I can write code in Pyspark. Wow, that's what a good explanatory should be like!

[@JacquelineSanchez-yh3dy]

I'm just getting started with creating a group CNN project with friends and we are dealing a huge dataset of mri scans so I was thinking about platforms that could deal with lots of data without having to deplete my disk lol. Thank you so much for breaking down how Apache works compared to Hadoop, I really appreciate it! 😊

[@elara5122]

What a video really understood the apache spark that i could not in my university.

[@Sky-2212]

Amazing, You explained everything in detail with examples. Best video on KZbin to know about Spark.👏

[@PriyanshuVerma-kv8lp]

I really understand the software really quickly, thanks man

[@hey_jaski]

crystal clear explanation! loved it❤

[@lamiranda100]

amazing explanation!! Thank you!

[@jeevanb8623]

superb man.. didn't waste the time.. great explaination..

[@ag564]

Very good intro to Spark. I've started my data science journey and it really helps.

[@JaydeepRana-j1c]

Thanks for this video , much informative and easy to understand using the examples you gave.

[@asp1201]

You nailed it man! Amazing information that i am using for my DE interviews

[@noob523]

Nice video

[@krishkanojia2850]

Understood video very well. Without any prior knowledge of apache spark

[@DarshilParmar]

Glad it was helpful

[@FarooqKhan-d8c]

Wonderfully explained in just 10 mins.

[@TahaSayyed-u2j]

One of the best video ! You really exxplained in very precise and esay way. Love it!

[@AayushSingh-gh5lm]

Clear and concise explanation

[@Kondaranjith3]

Waiting for full course from you apache spark

[@AnalyticsByHenry]

Impressive explanation of spark. Making it easy for every beginner to understand.

[@DarshilParmar]

Glad it was helpful!

[@krupakarjeeru1061]

You nailed it Bro in just 10 mins 😊

[@skshareena5013]

Super explanation bro, I got many answers in one vedio 🥳🥳

[@vijaybhargavkarnati9926]

Explained and the presentation is good.

[@venkatah9847]

Thank you very much and it's a very nice primer to refresh once the concepts. Thank you for your contributions 👍

[@sageevajoseph9579]

You explained the content simple and clear. Thank you for this video.

[@oo7ankit]

Brief and informative . Thanks 👍

[@mdaurangzebkhan8734]

A excellent video on Apache Spark. Covered almost everything. Very helpful video to the beginners like me.

[@2412_Sujoy_Das]

Darshil Sir, I had a query regarding Memory Management concept of Spark. As per my understanding, Spark uses it Execution memory to store intermediate data in execution memory which it shares with storage memory too, if needed. It can also utilize the off-heap memory for storing extra data. 1) Does it access the off heap memory after filling up storage memory? 2) What if it fills up Off heap memory too? Does it wait till GC clears up on-heap part or spills the extra data to disc? Now, in a wide transformation, Spark either sends the data back to disc or transfer it over the network, say for a join operation. Is the part of data sending data back to disc same as above where Spark has the option to spill data to disc on filling up on-heap memory? Please do clarify my above queries, sir. I feel like breaking my head as I couldn't make a headway through it yet even after referring few materials.

[@DarshilParmar]

In Spark, memory management involves both on-heap memory and off-heap memory. Let me address your queries regarding Spark's memory management: 1. Off-heap memory usage: By default, Spark primarily uses on-heap memory for storing data and execution metadata. However, Spark can also utilize off-heap memory for certain purposes, such as caching and data serialization. Off-heap memory is typically used when the data size exceeds the available on-heap memory or when explicit off-heap memory is configured. It is not used as an overflow for storage memory. 2. Filling up off-heap memory: If off-heap memory fills up, Spark does not automatically spill the data to disk. Instead, it relies on garbage collection (GC) to free up memory. Spark's memory management relies on the JVM's garbage collector to reclaim memory when it becomes necessary. When off-heap memory is full, Spark waits for the JVM's garbage collector to reclaim memory by cleaning up unused objects. Therefore, if off-heap memory fills up, Spark may experience performance degradation or even out-of-memory errors if the garbage collector cannot free enough memory. Thanks, ChatGPT

[@sophiebarrie5749]

Thank you so much for this explanation, youve outlined it quite clearly before Ive even had any experience using Spark, so thank you! If you could slow down your explanation a bit though, that would be helpful

[@DarshilParmar]

It's a 10min series, you can check out my courses for more in-depth guide

[@JonathanBrune]

Great introduction. Thank you so much.

[@hritikapal683]

Superb one! Can we expect full tutorial over spark!?

[@DarshilParmar]

Yes, coming soon!

[@hritikapal683]

@@DarshilParmar Ah nice then 😍

[@hobbyofbobby2973]

Good job Darshil. Appreciate the work.

[@kinjal_suryavanshi]

Thank you!! So helpful

[@rishisingh2598]

Fantastic explanation… 👏👏 the way you take your audience through the flow of explaining these concepts is very effective👌

[@DarshilParmar]

Thanks a lot 😊

[@ParthivShah]

Nice Video, Thank You.

[@prensudangol1547]

Thank u i got the basics

[@christinachen9669]

Wonderful summarize!

[@himanshusekharmohanta2489]

Really very nice explanations..

[@msanghamitra]

Very well explained! Thank you!

[@Player18345]

Super🎉 Waiting for full tutorial

[@DarshilParmar]

Very soon

[@jeanpeuplu3862]

Thank you for this video, I liked it: simple, clear, and short! Perfect :)

[@nancymaheshwari5421]

Just Amazing😇Thank you

[@darshakmehta]

To the point, quick, simple and comprehensive knowledge sharing!

[@DarshilParmar]

I appreciate that!

[@prashantcloud]

Very well explained , thank you very much

[@deepanshurathore9661]

You explained so many things in 10 minutes 🫡🫡🫡

[@DarshilParmar]

Thank you man

[@shankarchavhan375]

Darshil I want to learn data engineering from scratch. I don't know anything about these changes, so where do I start? Which course should be taken. 

[@DarshilParmar]

My Python & SQL for Data Engineering is a good place to start - learn.datawithdarshil.com/

[@lokeshnaidu6888]

Very well explained😊

[@arsh2489]

Thank you sir 👍

[@Taehyung-w3q]

really good explanation

[@vivekabhyankar5029]

Wonderful video you explained everything perfectly

[@garimajain474]

Best tutorial ❤❤all in one

[@mdmahirasefkabir3883]

Very nice video. Thank you!

[@imanelamnaoir590]

This explanation is very gooooooooooooooooooooooooood Thank u

[@DarshilParmar]

you are welcome mate!

[@hariramkm1677]

Excellent Explanation...

[@PranathiAnda]

Nice Explanation, Thank you

[@balajirpi]

As simple as that.. Liked

[@MrPavelber]

Great video! Thank you

[@ShrutiSharma-xu6qs]

very nicely explained

[@kartikeyasingh2798]

Very good video

[@TheBaBaLand]

Awesome video mate! well done.

[@nirmalpandey600]

Really productive video.

[@omarcruz6326]

Hello Darshil, This is great content ! A little bit too much information, hehe. Now it should be digested :)

[@karanjadhav2733]

Nicely presented and explained.

[@shantanukulkarni8883]

A very very good video. Thanks, you are doing a really great job!

[@InfinitesimallyInfinite]

Excellent video Darshil. Clear and concise! Subscribed!

[@ankushchavhan_]

Great explaination

[@Pvtmovies4384]

Thanks Darshil

[@ANKITASHARMA-ix9gt]

Very brief and informative video

[@paspas12]

Amazing concise detailed explanation with great editing. Such a great way of presenting a hard topic in an easy manner. Love your comparisons with teamwork, puzzles etc. So impressed. Big thumbs up and subscribe from me. Eager to see your other videos. Thanks!

[@aditya3david]

This is a great explanation

[@nik54610]

Great video bro

[@FrancisCarloA.Tadena-yn4jl]

THanks. very helpful.

[@vishalcrazy5121]

Really good content .

[@rk-ej9ep]

Nice explanation..plz do series on spark.

[@DarshilParmar]

I have a course on Spark, please check description

[@dineshkannant]

Nice one 👍

[@SivaKrishna-zj9jy]

Amazing content, keep up the good work, and thank you for the brilliant presentation. You really present topics precisely, simple-to-understand.

[@amadoucisse2692]

Nice video!

[@kirtisoni6076]

Amazing video. Please share the project doc😊

[@ericking4514]

nice job - short, to the point, great info. I really appreciate you sharing this. will like and subscribe.