01. Databricks: Spark Architecture & Internal Working Mechanism

#SparkArchitecture, #DatabricksArchitecture #Masterslave #DriverWorker #SparkExecutor #Spark Memory management #Sparkjobs #SparkRDD
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@souravchoudhury3698]

Not sure why your channel does not show while searching pyspark tutorial. I spoke to a developer on linkedin and he suggested me your channel. Great work thank you Sir!

[@rajasdataengineering7585]

Glad to hear it helps you! Thanks for visiting my channel

[@abhinavsingh2894]

This is an absolute masterpiece on introduction of Spark and all it's internal structure. Thank you for such a detailed video.

[@rajasdataengineering7585]

Thank you Abhinav👍🏻

[@abhinavsingh1173]

​@@rajasdataengineering7585 Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks

[@adithyabharadwaj5608]

Can beginners learn these?

[@PrashanthPatil-br8vj]

simple straight to the point absolute master class i was searching for this for long time no one taught it this easily thank you for this

[@rajasdataengineering7585]

Glad it helps! Thanks Prashanth, for your comment

[@hitvlogz]

Simple. Clear . To the point stuff. Thanks. Love your series.

[@rajasdataengineering7585]

Glad you like them! Thanks for your comment

[@figh761]

@@rajasdataengineering7585 Sir I would like to learn databricks fully .Please giude me

[@rajasdataengineering7585]

Pls go through all videos in this channel. You can learn databricks thoroughly

[@ravisaxena1599]

I really appreciate the way you have explained the difference between in memory computation and using external system.

[@rajasdataengineering7585]

Thank you Ravi

[@rudraganesh1507]

Yes great explanation

[@kushalthakkar5988]

​@@rajasdataengineering7585hi may you share PPT and databricks file of the course

[@merazshaik3504]

This series and explanation is too good than other channels and I still don't know why this channel is not showing any recommendation when we search for databricks vidoes.

[@rajasdataengineering7585]

Thank you 👍🏻

[@kasmitharam982]

To the point and crisp yet detailed explanation, I've seen in a while, thank you so much!

[@rajasdataengineering7585]

Glad it was helpful!

[@shivanaga3302]

Best intro of spark which i've seen till now.......

[@rajasdataengineering7585]

Thank you

[@boseashish]

you have put in a lots of effort and tried to cover all important points. thank you very much for your immense contributions

[@rajasdataengineering7585]

My pleasure! Thank you for your comment

[@sunilt1739]

Thank you so much for putting such a great effort. I haven't gone thru all videos yet, but i can definitely imagine the hard work that you must have put behind this playlist.

[@rajasdataengineering7585]

Thank you so much!

[@prathapganesh7021]

I have searched lots of videos regarding spark arictecture and working but this is videos is awesome I realy appreciate for this video nice presentation and i understand very clearly complete concepts thank you so much🙏🙏

[@rajasdataengineering7585]

Glad it was helpful! Thanks for your comment

[@KarthikChavan-zs7iz]

Just started but I love clear and simple explanation, thanks a lot for your efforts

[@rajasdataengineering7585]

You're very welcome! Glad it helped

[@kumarashirwadmishra7414]

Thanks Sir for Wonderful Explanation and provided in-dept Knowledge of Spark Architecture. Wonderful Resource for start SPARK Journey.

[@rajasdataengineering7585]

Thanks and welcome

[@sangeethaezhumalai168]

Appreciate your detailed explanation Sir... Really helpful

[@rajasdataengineering7585]

Glad it is helpful. Thanks for your comment

[@vutv5742]

Superb , Fantastic , Marvellous.....What a great teacher you are .

[@rajasdataengineering7585]

Thank you so much! Glad it helps

[@vutv5742]

Yes with clarity you have explained architecture and specially the partitioning with diagram was really helpful.@@rajasdataengineering7585

[@charangowdamn8661]

Hi sir do you conduct any online coarses

[@debasishkalia135]

this explanation is great , very detailed

[@rajasdataengineering7585]

Thank you!

[@ranyasri1092]

Thanks alot for in depth explanation😊

[@rajasdataengineering7585]

Hope it helps! Thanks and welcome

[@sowjanyagvs7780]

when you mention referring the other videos, can you also keep mentioning those links in description. Thanks a lot for your explanation!!

[@rajasdataengineering7585]

Sure thing! Will add links

[@ashswinsubbiah3752]

What an explanation, thank you so much sir.

[@rajasdataengineering7585]

You are most welcome

[@MindBodyEvolutionTV]

Thank you for great and fantastic master pieces

[@rajasdataengineering7585]

Thanks for listening!

[@mayank113463]

excellent exp;anation across all the youtube channels thanks

[@rajasdataengineering7585]

Much appreciated! Thanks for your comment

[@karthiknani1503]

Thankyou very much for the content Sir.

[@rajasdataengineering7585]

Glad it helps you gaining insight about spark internals!

[@karthikeyana6490]

Just starting to watch yr playlist with the hope to learn spark, lets see how it goes. BTW thaks for the complete playlist mate!

[@rajasdataengineering7585]

Hope you enjoy it!

[@HarshaVardhan-ox2zh]

Thank you for making vedios Actually helped a lot.....

[@rajasdataengineering7585]

Thanks Harsha, for your comment!

[@harikareddy579]

Amazing explanation sir, I am able to understand it very clearly

[@rajasdataengineering7585]

Thanks. Glad you enjoyed this content!

[@bashaali1685]

​@@rajasdataengineering7585 hi sir can i talk to you ..can i get ur contact num plzzz

[@Moeistic]

Very well explained Raja, thanks for making this series brother.

[@rajasdataengineering7585]

Thanks Nasser

[@divyamariyameldo6495]

Thanks for the content!

[@rajasdataengineering7585]

My pleasure! Welcome

[@vydudraksharam5960]

Raja, this is excellent way of explanation .

[@rajasdataengineering7585]

Thank you, Vydu!

[@arunshankar1987]

Exactly what am looking for. Please let me know where I can find the datasets to practice.

[@pridename2858]

Yes, this is master piece. Thanks

[@rajasdataengineering7585]

Welcome, Glad you like it!

[@Sreenivasan-cn5qv]

for sure best video ever seen before... Raja Great Presentation

[@rajasdataengineering7585]

Glad you liked it! Thanks for your comment

[@varun8952]

Hi Raja, This is a great explanation. Appreciate your hard work.

[@rajasdataengineering7585]

Thank you!

[@vidhikumar1664]

Great explanation.

[@rajasdataengineering7585]

Glad it was helpful!

[@naveenraj9977]

Very good explanation,I watched all your playlist gain knowledge about spark and writing code also,I hope to do more vedios on spark , I'm requesting you to upload vedios with subtitles too so we can make a note of the entire session, please add subtitles too for you old vedios.

[@rajasdataengineering7585]

Thanks Naveen! Sure, will try to add subtitles

[@naveenraj9977]

@@rajasdataengineering7585 Thanks,that will be great

[@RamiReddy-y8r]

your explanation very excellent

[@rajasdataengineering7585]

Glad it was helpful!

[@rahulmittal116]

Excellent video

[@rajasdataengineering7585]

Thank you very much!

[@terrificmenace]

Excellent video 😀 thank you

[@shusantsapkota9871]

Could you please provide the slides used in all the lectures. This will be super useful. Thank you for this master pieces!!.

[@adig8881]

Watch in full screen, and take screenshots bro..

[@SystemTinu]

Great video Raja!! Explained very well..Thanks

[@rajasdataengineering7585]

Thank you

[@saimounika6475]

great explaination sir

[@rajasdataengineering7585]

Thanks! Hope you find it helpful

[@AnandKumar-dc2bf]

nice pictorial representations bro keep gng

[@rajasdataengineering7585]

Thanks Anand

[@shekarsubramani9861]

Hi Raja, I was very much confused with the architecture, once I saw your video ,now its clear, Keep up the good work

[@rajasdataengineering7585]

Thanks Shekar!

[@subhashyadav9262]

Very Nice

[@rajasdataengineering7585]

Thanks

[@dineshbvbv6479]

Good explanation! keep up the good work.

[@rajasdataengineering7585]

Thank you

[@vinothloganathan2623]

Hi Raja, One of the amazing explanation. I couldn't find these level of details in any of the source like - books, medium and other youtube. Amazing work !!. Could you share if there are any resource helped you for spark >

[@rajasdataengineering7585]

Thank you! I don't have any other resources. I summarised these concepts based on my working experience

[@pralgs628]

Awesome!! Could you please attach the PPT for Each Video.. Thanks

[@ayanjit9196]

Sir can you please give us the links of the notebooks used in this series. This has helped me and a lot of other people. Giving this link would be even more helpful 🙏🙏🙏

[@100useful7]

Yes, please

[@krishnamurthy1243]

Hi Raja ,please do azure synapse analytics,eagerly waiting

[@rajasdataengineering7585]

Sure Krishna, will create videos on synapse analytics

[@tanushreenagar3116]

perfect video sir

[@rajasdataengineering7585]

Thank you!

[@abinashsenapati1880]

It is really helpful. Thank you.. Where will I get the complete PPT of this playlist?

[@anilmnt82]

I do see it as more detailed explanation on spark , but not really on Databricks, missing many Databricks features like Unity catalog, DBFS, Vaccum, Liquid clustering etc..

[@AnandKumar-dc2bf]

Nice explanation...

[@vishalaaa1]

Excellent

[@rajasdataengineering7585]

Thank you! Cheers!

[@JindamSrilekha]

Well Explained

[@rajasdataengineering7585]

Thanks

[@sunitachoudhary6348]

Very good course🎉

[@rajasdataengineering7585]

Glad you think so! Thanks

[@rudraganesh1507]

This is the masterpiece

[@rajasdataengineering7585]

Thanks Ajay

[@saravninja]

Started your videos!! All are great

[@rajasdataengineering7585]

Thank you Ninja

[@saravninja]

@@rajasdataengineering7585 I went through complete video second by second. Video has lot of clarity than any other KZbin channel. Keep up good work!! Have you experienced data skew issue, if yes can you point video or do video for us.

[@rajasdataengineering7585]

Thank you for your kind words. It gives a motivation to create more videos which can help genuine knowledge seekers like you. For data skew, have posted one video (though it does not cover advanced concepts) kzbin.info/www/bejne/e4LLnZevgbyDras

[@saravninja]

@@rajasdataengineering7585 thanks a lot again Raja!! Will go through it. I am looking for airflow training, I have sent mail to you, kindly respond.

[@CoopmanGreg]

Great Video! 👍

[@rajasdataengineering7585]

Thanks 👍🏻

[@shamsmalek]

Excellent job. Can you please provide me the data set and code? Or please give me the Git link to download the dataset and code for your tutorials. Thanks.

[@shyamkumardhamode4475]

Soooo good explanation

[@rajasdataengineering7585]

Glad it was helpful!

[@alex45688]

good explanation

[@rajasdataengineering7585]

Thanks for liking

[@gulsahtanay2341]

Thank you!

[@rajasdataengineering7585]

Thanks for your comment!

[@amrutavastrad]

Thank you!!!

[@rajasdataengineering7585]

You're welcome!

[@mohammedmujahiduddin4715]

Thank you Raja for the detailed explanation. Do we have any video which is focusing on Worker Node and its details ? And as you were about to make a video regarding the memory management details, please also share that or the video title if already present. Thank you so much in advance!

[@rajasdataengineering7585]

Please watch videos kzbin.info/www/bejne/mYXNeaKhpN1of9U kzbin.info/www/bejne/d2mToGyNfL1-las

[@NikhilGosavi-go7be]

done

[@jayaprakashm2849]

Nice info

[@neelbanerjee7875]

Thank you ver much for such contents.. one request - Can you please make a video on real time executor number, core, memory allocation based on input data size like. 1. 1-5 gb 2. 5-15 gb 3. 15-25 gb 4. 25-50 gb 5. > 50gb = 1 tb

[@rajasdataengineering7585]

Sure Neel, will make a video on this requirement

[@mohitupadhayay1439]

@@rajasdataengineering7585 Waiting for this!

[@VipinYadav-ii1ow]

Just starting to learn spark and databrics. Is this resource is enough to crack entry level data engineering job?

[@rajasdataengineering7585]

Yes definitely, these videos are more than enough to crack entry level job

[@sudippandit9855]

excellent explanation!!

[@rajasdataengineering7585]

Thank you Sudip

[@ElhamMirshekari]

Raja could you kindly make a video on these three functions and compare them: Join, Union, Concat

[@rajasdataengineering7585]

Hi Ellie, I have already created videos for join and union. Will make a video for concat as per your request. Join : kzbin.info/www/bejne/pHuqm3mDhaefisk Union: kzbin.info/www/bejne/fIW3fYB4gc6tjJo hope it helps you

[@ranjansrivastava9256]

How was 50 Partitions per executor has been calculated Raja?

[@siddavatamvenugopalreddy9686]

Is this all tutorials related to spark only? Or it includes data bricks aswell? Please confirm

[@rajasdataengineering7585]

It's more on databricks which is internally using apache Spark

[@kneelakanta8137]

can I have document for reference of this playlist

[@srikanthbachina7764]

HI Raj, Videos are Missing from 27 to 30 Could you Please Upload them.

[@rajasdataengineering7585]

Hi Srikanth, those 4 videos are related to Azure Synapse analytics. Its still available under all videos section

[@mihirpatel2512]

Nice Videos! can you share the slides?

[@arupnaskar3818]

Hi Raja u r teaching is awesome .... really help .. sir, just wanted to know .. for "STAGING" here u mentioned about "Nodes" .. here "Nodes" means No of worker Nodes or Partitions ??

[@rajasdataengineering7585]

Thank you Arup. Node means no of worker nodes in the cluster

[@arupnaskar3818]

@@rajasdataengineering7585 thanks raja ..💐🌸

[@shivayogihiremath4785]

I'm following this channel from couple of days now. The content and way of explanation is awesome. Good job my friend. keep up the good work. wishing you all the very best. one small suggestion, if possible, please try to avoid the initial music (which is played at the beginning of the video) at times it is annoying. thank you!

[@rajasdataengineering7585]

Hi Shiv, thank you for your valuable comments. I already removed this initial music. May be it is still there for only few initial videos.

[@morgann4276]

Great video raja!! Wanted to know how you have such in depth knowledge.. did you learn from spark docs ?

[@rajasdataengineering7585]

Thanks Morgan. Yes spark documents and working experience helped me to understand concepts

[@oluakano6497]

hi, i am new to spark and your videos seem like a great resource to learn. i am wondering what is the best order to watch them? through the playlist pr just use the numbers like 1,2,3...

[@rajasdataengineering7585]

Hi, for all videos, I have given serial number. You can follow the order based on that serial number

[@samant6]

there are some videos missing? like 27 , 28, 29 etc?

[@dipanjanpan1]

Can we create multiple executor node on a worker node?

[@rajasdataengineering7585]

Yes we can. Executor is logical division of computing resources

[@Arun-uw1hy]

Hi, here the executor mean processor (CPU) , Right? Because each node may have multiple CPU, and also each CPU can have multiple cores. So each cores has a separate executor.

[@rajasdataengineering7585]

Executor means logical division of nodes. That means combination of processor + memory + network

[@sravanthiyethapu9970]

Hi Raja,I want to learn databricks for azure data engineer. Will this playlist help me for interview??

[@rajasdataengineering7585]

Hi Sravanthi, yes this playlist definitely helps you to perform well in interview

[@BlingKing321]

In order to store data in JVM memory we need to do serialization and deserialization. Why ?

[@Umerkhange]

FROM WHERE TO START? IS THERE ANY SEQUENCE TO FOLLOW??

[@rajasdataengineering7585]

Playlist is already sorted in correct order. You can follow the playlist

[@NA-dg6um]

Can you provide documentation of this video

[@البداية-ذ1ذ]

please could you make videos in examples about pyspark in real projects

[@rajasdataengineering7585]

Sure, will make videos on real time projects

[@البداية-ذ1ذ]

Thanks alot

[@aashishmalhotra]

amazing

[@rajasdataengineering7585]

Thank you! Cheers!

[@gautampoddar3392]

What is the total duration of the playlist

[@ParthKhambhayta-dj9te]

Sir you said when read CSV file it's divided in default 200 partitions but default size of block is 128MB so it should decide into 16 partition please let me know am I correct or not ?

[@manojru1]

I have installed Jupyter with Pyspark...where should I run my command to see the Spark job like you are showing on 38:21sec? or should I install some other IDE for that?

[@mannykhan7752]

Yun number of worker nodes?? What is Yun or yum??

[@dinesh_tadepalli]

Are any prerequisites required to this pyspark series?

[@rajasdataengineering7585]

No, nothing needed. I have covered from basic

[@TelugupodcasterDT]

Thank you!

[@dataeanalytics]

I have reacted this video on my youtube channel to people in Brazil who don’t speak English 😂😂

[@tushargp]

Hi , is there any email to directly reach out for questions ?

[@pradeepkumarg.c3563]

pls share the road map for azure databricks, if possible

[@bhawna927]

inside the worker node how many executor will be there who will decide this

[@saikrishna1939]

Can you please guide me how to start your videos I mean the order I can see many playlists in the channel. I want to learn spark and data bricks

[@rajasdataengineering7585]

Sure bro, let me give serial number to my videos so that you can follow the structured learning list

[@saikrishna1939]

@@rajasdataengineering7585 yeah thanks, also please comment here that which playlists we shld follow for the order to learn spark and data bricks 🙂

[@rajasdataengineering7585]

Sure

[@rajasdataengineering7585]

@@saikrishna1939 The videos are given with serial number. You can follow with that sequence

[@saikrishna1939]

@@rajasdataengineering7585 yes but which playlist to follow as there are 7 playlists in the channel so it's Lil confusion. Say for example a playlist has 5 videos but when I open it I can see 17, 18, 19, 20 in the videos. In interview series it starts with 1 again

[@shravyakulal5756]

Hi Do you offer any course private in detail? with real time project

[@nitin1808]

Snowflake and Databricks is almost the same...?