spark overview | lec-1

From this video, we are going to start talking about Apache spark in great depth.
Directly connect with me on:- topmate.io/manish_kumar25
For more queries reach out to me on my below social media handle.
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj

[@udaypratap539]

I have cleared 'Databricks Certified Associate Developer for Apache Spark' certificate yesterday but will still follow this playlist because I like your explanation. You're doing great work🙏🙏.

[@ayushtiwari104]

The fact that you answer to all the "why" and "what" of a simple definition makes learning so much in depth and enjoyable! Great teaching!

[@boseashish]

sir, aap ne hum sab ke liye phir se video banaya hai...hum sab ke liye bahut he achhi baat ho gai...

[@dhwanimehta4634]

heads off to your honesty Manish for each video lecture you deliver.

[@machhindrasmathematics2551]

Nice to see with spark...sir

[@mynameisjimmy9823]

Bahut badiya!! Kya baat hain. You are so good Man!!

[@mayankdubey7477]

I have seen so many english tutorial for spark but the way you are explaining is just amazing. Content are more clear and easy to understand. Keep up the good teaching work. Hats off 👏

[@UnKnown-id7ih]

Great work bhiya

[@adityasawarkar2248]

keep making vdos brother ......ur every vdo outclass other so called data techs

[@user-mo7nf7yi8x]

nice explanation as well as apka sikhate samay bahut hi sahajeta se samjhana achha laga .pl. continue this simplicity hame encourage karti hai. thanaks Sir..🙏🙏🙏gr8 work...

[@SQLwithManoj]

Loved the "baap & 4-beta" analogy... gr8 content !!!

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@kalpeshmahajan5552]

nicely explained,no videos from almost 3 weeks, waiting for next

[@shekharchintalwar1272]

❤ your content

[@ashishlimaye2408]

Great video !!

[@amolmanorkar2841]

Thanks a lot for this!, this is really helpful :)

[@HarshitSingh-it7kp]

Hey Manish. Really appreciate your efforts. These are gem lectures man.

[@ArpitSharma-kk9tl]

Thanks bro, sharing info

[@lazycool3611]

examples ache hai smjhne ke liye

[@pratikparbhane8677]

Attendance Marked

[@mohitkeshwani456]

Excellent

[@mukundraj4021]

super!!

[@gchanakya2979]

Thankyou so much bhaiya 😭

[@navjotsingh-hl1jg]

very good explanatiom

[@shivarajhalageri2513]

Excited for upcoming next video s❤

[@manish_kumar_1]

Will upload soon

[@gopalsharma-yz2ti]

Bro I am from Patna working as a web scraper in b2b company . I can assured u have millions subscriber one day …… keep going best of luck

[@gautamKumar-dg3ss]

very nice

[@ManishSharma-fi2vr]

❤

[@mohammadfurquan241]

Thank you waiting for more

[@manish_kumar_1]

Coming soon

[@dhruvnayak7922]

Very nicely explained. Thank you Bhaiya.

[@vijibang]

How it's possible to comment before 10days when video is released today.

[@nobody-789]

I have decided to follow your channel, funny thing is i am poor in hindi so, i don't know if I'll learn DE. Most probably I'll end up learning Hindi. Anyway looking forward for more videos on how to crack DE .

[@ujjalroy1442]

Awesome explanation bro.... Must continue the series

[@manish_kumar_1]

Yes I will upload video soon

[@AmitSharma-ow8wm]

waiting for ur next vidio..😊

[@manish_kumar_1]

It will come today

[@Rahul-wu1ig]

bhai ye videos continue rkhna.

[@manish_kumar_1]

Bilkul

[@amansoniamanat]

You are very great person bro 🙏

[@manish_kumar_1]

Thank you so much 😀

[@krishnashaw6766]

Thank you so much for great content and explanation. can we get pdf notes of all these theories?

[@nitilpoddar]

done

[@mukundraj4021]

please start soon, waiting

[@manish_kumar_1]

Already started sir

[@tusharagarwal3553]

Hello, when you will be creating playlist for Hadoop. Please make

[@Astha.Sharma1]

Spark doesn’t have its own storage but in the computer cluster architecture explanation it’s mentioned that each slave has certain RAM, CPU and Storage. What does storage meant here?

[@coding7241]

very nice class ...looking for full series

[@manish_kumar_1]

Will upload soon

[@ishanrazdan]

​​@@manish_kumar_1bhai can u plz make how to read spark ui whenever spark job is slow , kase identify karenge , not able to get proper video Optimization playlist bna skte ho mostly asked in interview

[@Somnath-je9nd]

thanks manish!!

[@manish_kumar_1]

Stay tuned for the series 🙂

[@Somnath-je9nd]

@@manish_kumar_1 I have experience in support domain. but now looking to switch into data domain. hope your series helps me.

[@manish_kumar_1]

@@Somnath-je9nd Definitely it will

[@codewithovi997]

Thank you. This means that the storage or disc is irrelevant in spark cluster

[@manish_kumar_1]

Aisa nhi hai. Aapko storage to lagega data ko permanently store karne ke liye. Aapke paas option bahut hai ki aap ise kaha rakhna chahte ho like hdfs, cloud or your on premise server

[@codewithovi997]

@@manish_kumar_1 g thek hai. Means ky spark itself ka storage sy koi laina daina nae hai but hum spark ky sath different storage services use kr sakty hain. Thank you

[@tanmaykapil5362]

bhaiya is both of your series enough for interviews?

[@user-wg4bh3rv5i]

what is configuration

[@Mahesh717]

Hello sir, I had learnt sql, power bi & python for Data analyst role. but as of now, there are very less job for freshers in this field, what about data enginner I dont have any idea in this field is there are jobs for fresher in DE....

[@manish_kumar_1]

Idhar to Data analyst se v Kam openings hote hai. But few freshers get chance to work as DE directly. Openings are bit less compared to Data analyst

[@satyamrai2577]

Namaste Manish bhaiya. Please see this series through to the end. Hope to learn a lot from you. 🙏

[@manish_kumar_1]

Yes sure

[@lifelearningneo]

achha bhaiya agar hamare cluster ka size kam baa aur agar usase jyada size ka data aa gaya tab ye kaise handel karta hai...aur data streaming hai..

[@manish_kumar_1]

Streaming ka bilkul idea nahi hai mere ko

[@aditya9c]

how to learn both spark overview playlist and spark practical playlist. Pahle kaun sa padhe ? ya agar sath me padhna h to kya flow rahega .

[@manish_kumar_1]

After 5-6 videos of fundamentals you can start practical in parallel

[@roshan_off1955]

Bhai ek resume ka video banado 2 years dusre stream se data engineer k liye

[@manish_kumar_1]

Ok

[@surojitsengupta5162]

spark data store nhi karta. Lekin shuffling me jo disk I/O hota hai, us data ke sath kya hota hai..post execution?

[@manish_kumar_1]

Clean kar diya jata hai

[@mohdhammadkhan5570]

Has anyone made notes.

[@wellwisher7333]

Sir, this playlist would have totally how many lectures, approx?

[@manish_kumar_1]

Around 20-25 for experienced and 12-15 will be enough for Beginner

[@wellwisher7333]

@@manish_kumar_1 Thank you Sir

[@RaNd0MiZeD]

Main leta hi hua tha 😂

[@manish_kumar_1]

😂😂

[@roshan_off1955]

Bhai syllabus spark ka refer krdo ,proper mil nhi rha h

[@manish_kumar_1]

Spark internals Spark submit Spark process life cycle Spark optimization Data frame API coding

[@mukundraj4021]

aap course start kb kroge data engineer ka

[@nilavnayan4521]

Yahi hai course bro - and this is lec 1 of series

[@Binge_View]

Storage nahi hota but architecture me aap bol rahe storage 5TB, yeh confusion please clear kare

[@manish_kumar_1]

Spark ka khud ka nhi Hota hai. Agar storage nhi hoga to Data persists kaise hoga.

[@Binge_View]

@@manish_kumar_1 data persistence RAM me hona chahiye, Jaisa aapne bola yahi toh diffrence hai Hadoop and Spark me, kya aap example bata sakte jab data HDD/SDD me store hoga instead of RAM.

[@manish_kumar_1]

@@Binge_View you need to work a lot on understanding fundamentals of computer science. RAM short lived memory hai.

[@Binge_View]

@@manish_kumar_1 Sir, I have done B.E in computer science, I understand RAM is temporary memory, HDD we can use as heap memory or for data view swapping, even I know how can we see 4GB file with 1GB RAM and some more in depth details about it, what iam saying is Hadoop Map Reduce used to work like, whenever data transformation come to it, it will perform some transformation and store the result to HDFS and then perform another transformation and as required fetch the previous data from HDFS, as HDFS is slow in reading and writing spark introduced. What I am not understanding is, if spark also stores data in HDD then in which case it's happening and how it's different from Hadoop. Please don't take anything in a negative manner, I really like your way of explaining and clearing the concept, you are the best teacher I found for spark concepts. I hope you understand.

[@amansoniamanat]

Bro , how much coding knowledge is required for data engineer

[@Venom-yk3wu]

python

[@manish_kumar_1]

Array and string ke easy and medium level ka questions kijiye from leetcode. Itna me kaam ho jayega

[@amansoniamanat]

@@manish_kumar_1 and python pura aana chahiye ? Ya basic concept sufficient hai

[@manish_kumar_1]

@@amansoniamanat syntax aana chahiye like loops, variables,list,dictionary,function and basics of Oops concept

[@amansoniamanat]

@@manish_kumar_1 okay, Thank you bro 🙏