spark ecosystem | Lec - 4

In this video, I have talked about all the component of saprk. And how it stich to all of the component to run an spark application.
Directly connect with me on:- topmate.io/man...
For more queries reach out to me on my below social media handle.
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj

[@bhavyamalviya8364]

Wonderful theoretical explanation

[@nilavnayan4521]

Manish bhai, watched all your last 4 videos (this series) in one sitting and loved it - waiting for more :) Edit - Not just watched laet ke - but copy kalam ke saath :D

[@HanuamnthReddy]

Bhai can you do databricks

[@andekhiaawaz]

Zabardast Series

[@kavyabhatnagar716]

Great series ❤

[@sairamguptha9988]

Hi Manish, you are doing an awesome job... Also pls try to do more vidoes on spark practicals like how to practice spark on data bricks account.

[@manish_kumar_1]

Already doing that

[@sanooosai]

thank you sir

[@Mitra-xb7tz]

So, can we use HDFS cluster upon which Spark will act for data processing ?

[@Watson22j]

yes we can

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@kartikmod7952]

Hi, As per the video if the situation occurs when the cluster manager is down then the request will not go further, is this a possible case?

[@manish_kumar_1]

Yes. If your cluster is down then your request won't reach till the server. Master machine(on which yarn runs) are more powerful than the workers node and also it has a standby machine. So it is highly unlikely that your master will be down

[@gchanakya2979]

Bhaiya dsa python me hi karu ya fir muje java ya c++ me karna chahiya. Apne kya kiya.

[@manish_kumar_1]

You should do either in python or java. Because DE me ye 2 language hi use hoti hai

[@alakanandade3883]

@manish_kumar_1 does hadoop create replication of data for intermediate data as well? And how are these deleted later on?

[@ankitsrivastava6218]

Hadoop intermediate data ka replication nahi karta across multiple nodes, jaise ki wo stored data (HDFS data) ke saath karta hai. Intermediate data sirf temporarily local disk pe store rehta hai jahaan Mapper run ho raha hota hai. Agar koi Mapper fail ho gaya toh Hadoop us Map task ko automatically doosre node pe re-execute karta hai, aur phir se wo intermediate data generate hota hai.

[@praveenkumarrai101]

bro you said we will use dataframe and spark sql then what is the use of python ? I am really confused.

[@manish_kumar_1]

To stich all your components together you need python. Let say I ask you to get the data from s3 bucket and make some changes on path and then transform it. So in this example you will use programming language I.e python in our case to access the file and make adjustment in the path. And then that path will be passed as a parameters to spark. In this playlist we are just learning spark

[@praveenkumarrai101]

@@manish_kumar_1 ok brother thanks

[@saumyasingh9620]

Your videos are great...Can you pls make these little longer, capturing more concepts in one videos.

[@manish_kumar_1]

Yes I will

[@nitilpoddar]

done

[@navjotsingh-hl1jg]

manish bhai what are the roles of executor? executor means execute the data am i right ? or what is driver bro?

[@younevano]

Driver orchestrates the job and Executors do the data processing tasks!

[@explorewithaAB]

Since spark is open source, so who does pay for this hardware. Since I think that hardware is nothing but a physical machine so there has to be someone who is paying right?

[@manish_kumar_1]

Spark is a framework that is nothing but a piece of code which helps you to run your Data processing work in distributed manner. To run your code you need a cluster that is nothing but a physical machine

[@explorewithaAB]

@@manish_kumar_1 thanks a lot for the reply. And who does pay for this cluster as it is yet to deploy on cloud service

[@sankuM]

@@explorewithaAB , I believe spark is just a frame work developed in Open source manner which will allow you to utilize distributed processing over cluster of compute(rs)! The cost of any such cluster is paid by the users only. e.g., if you run spark on AWS EMR or Databricks, you're charged money as per your usage of compute resources! Hope this helps! I'm exploring databricks community edition for spark practice, and am unsure as to how those clusters are managed & paid for internally! @Manish Kumar

[@bangalibangalore2404]

​@@explorewithaABif you install on laptop then your laptop you are buying, companies initially used on premise servers which they had to pay and purchase but now companies go to Azure, Azure gives them the computer on rent remember this way, the more powerful processor and hard disk you need the more you have to pay, at the end of month you will receive a bill for the usage that you did

[@ordinary_indian]

Well explained

[@nitilpoddar]

done