01. Databricks: Spark Architecture & Internal Working Mechanism

[@souravchoudhury3698]

Not sure why your channel does not show while searching pyspark tutorial. I spoke to a developer on linkedin and he suggested me your channel. Great work thank you Sir!

[@rajasdataengineering7585]

Glad to hear it helps you! Thanks for visiting my channel

[@jatinsethi6415]

Thanks for the videos. Your videos really helped me to switch my job. Thanks for the great content. Your explanation is awesome. Thanks again.

[@rajasdataengineering7585]

Great to hear! Thank you so much!

[@abhinavsingh2894]

This is an absolute masterpiece on introduction of Spark and all it's internal structure. Thank you for such a detailed video.

[@rajasdataengineering7585]

Thank you Abhinav👍🏻

[@abhinavsingh1173]

​@@rajasdataengineering7585 Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks

[@adithyabharadwaj5608]

Can beginners learn these?

[@hitvlogz]

Simple. Clear . To the point stuff. Thanks. Love your series.

[@rajasdataengineering7585]

Glad you like them! Thanks for your comment

[@figh761]

@@rajasdataengineering7585 Sir I would like to learn databricks fully .Please giude me

[@rajasdataengineering7585]

Pls go through all videos in this channel. You can learn databricks thoroughly

[@merazshaik3504]

This series and explanation is too good than other channels and I still don't know why this channel is not showing any recommendation when we search for databricks vidoes.

[@rajasdataengineering7585]

Thank you 👍🏻

[@ravisaxena1599]

I really appreciate the way you have explained the difference between in memory computation and using external system.

[@rajasdataengineering7585]

Thank you Ravi

[@rudraganesh1507]

Yes great explanation

[@kushalthakkar5988]

​@@rajasdataengineering7585hi may you share PPT and databricks file of the course

[@shivanaga3302]

Best intro of spark which i've seen till now.......

[@rajasdataengineering7585]

Thank you

[@vutv5742]

Superb , Fantastic , Marvellous.....What a great teacher you are .

[@rajasdataengineering7585]

Thank you so much! Glad it helps

[@vutv5742]

Yes with clarity you have explained architecture and specially the partitioning with diagram was really helpful.@@rajasdataengineering7585

[@charangowdamn8661]

Hi sir do you conduct any online coarses

[@PrashanthPatil-br8vj]

simple straight to the point absolute master class i was searching for this for long time no one taught it this easily thank you for this

[@rajasdataengineering7585]

Glad it helps! Thanks Prashanth, for your comment

[@kumarashirwadmishra7414]

Thanks Sir for Wonderful Explanation and provided in-dept Knowledge of Spark Architecture. Wonderful Resource for start SPARK Journey.

[@rajasdataengineering7585]

Thanks and welcome

[@ankitachaturvedi1138]

This interview series is really helpful. I haven't worked much on databricks but these videos are giving great insight of internal working & concepts. I am able to crack interviews. Thanks a lot for such informative videos!!

[@rajasdataengineering7585]

Glad to hear this! Keep watching

[@prathapganesh7021]

I have searched lots of videos regarding spark arictecture and working but this is videos is awesome I realy appreciate for this video nice presentation and i understand very clearly complete concepts thank you so much🙏🙏

[@rajasdataengineering7585]

Glad it was helpful! Thanks for your comment

[@sunilt1739]

Thank you so much for putting such a great effort. I haven't gone thru all videos yet, but i can definitely imagine the hard work that you must have put behind this playlist.

[@rajasdataengineering7585]

Thank you so much!

[@boseashish]

you have put in a lots of effort and tried to cover all important points. thank you very much for your immense contributions

[@rajasdataengineering7585]

My pleasure! Thank you for your comment

[@KarthikChavan-zs7iz]

Just started but I love clear and simple explanation, thanks a lot for your efforts

[@rajasdataengineering7585]

You're very welcome! Glad it helped

[@sangeethaezhumalai168]

Appreciate your detailed explanation Sir... Really helpful

[@rajasdataengineering7585]

Glad it is helpful. Thanks for your comment

[@vydudraksharam5960]

Raja, this is excellent way of explanation .

[@rajasdataengineering7585]

Thank you, Vydu!

[@Sreenivasan-cn5qv]

for sure best video ever seen before... Raja Great Presentation

[@rajasdataengineering7585]

Glad you liked it! Thanks for your comment

[@kasmitharam982]

To the point and crisp yet detailed explanation, I've seen in a while, thank you so much!

[@rajasdataengineering7585]

Glad it was helpful!

[@mayank_om]

excellent exp;anation across all the youtube channels thanks

[@rajasdataengineering7585]

Much appreciated! Thanks for your comment

[@rk-ej9ep]

Such a great info. Awesome..

[@rajasdataengineering7585]

Glad it was helpful! Keep watching

[@debasishkalia135]

this explanation is great , very detailed

[@rajasdataengineering7585]

Thank you!

[@Moeistic]

Very well explained Raja, thanks for making this series brother.

[@rajasdataengineering7585]

Thanks Nasser

[@ashswinsubbiah3752]

What an explanation, thank you so much sir.

[@rajasdataengineering7585]

You are most welcome

[@harikareddy579]

Amazing explanation sir, I am able to understand it very clearly

[@rajasdataengineering7585]

Thanks. Glad you enjoyed this content!

[@bashaali1685]

​@@rajasdataengineering7585 hi sir can i talk to you ..can i get ur contact num plzzz

[@RamiReddy-y8r]

your explanation very excellent

[@rajasdataengineering7585]

Glad it was helpful!

[@ranyasri1092]

Thanks alot for in depth explanation😊

[@rajasdataengineering7585]

Hope it helps! Thanks and welcome

[@karthikeyana6490]

Just starting to watch yr playlist with the hope to learn spark, lets see how it goes. BTW thaks for the complete playlist mate!

[@rajasdataengineering7585]

Hope you enjoy it!

[@MindBodyEvolutionTV]

Thank you for great and fantastic master pieces

[@rajasdataengineering7585]

Thanks for listening!

[@ShivamGupta-wn9mo]

great playlist

[@rajasdataengineering7585]

Thank you

[@karthiknani1503]

Thankyou very much for the content Sir.

[@rajasdataengineering7585]

Glad it helps you gaining insight about spark internals!

[@varun8952]

Hi Raja, This is a great explanation. Appreciate your hard work.

[@rajasdataengineering7585]

Thank you!

[@sowjanyagvs7780]

when you mention referring the other videos, can you also keep mentioning those links in description. Thanks a lot for your explanation!!

[@rajasdataengineering7585]

Sure thing! Will add links

[@saimounika6475]

great explaination sir

[@rajasdataengineering7585]

Thanks! Hope you find it helpful

[@SystemTinu]

Great video Raja!! Explained very well..Thanks

[@rajasdataengineering7585]

Thank you

[@rahulmittal116]

Excellent video

[@rajasdataengineering7585]

Thank you very much!

[@HarshaVardhan-ox2zh]

Thank you for making vedios Actually helped a lot.....

[@rajasdataengineering7585]

Thanks Harsha, for your comment!

[@shusants]

Could you please provide the slides used in all the lectures. This will be super useful. Thank you for this master pieces!!.

[@adig8881]

Watch in full screen, and take screenshots bro..

[@dineshbvbv6479]

Good explanation! keep up the good work.

[@rajasdataengineering7585]

Thank you

[@sunitachoudhary6348]

Very good course🎉

[@rajasdataengineering7585]

Glad you think so! Thanks

[@naveenraj9977]

Very good explanation,I watched all your playlist gain knowledge about spark and writing code also,I hope to do more vedios on spark , I'm requesting you to upload vedios with subtitles too so we can make a note of the entire session, please add subtitles too for you old vedios.

[@rajasdataengineering7585]

Thanks Naveen! Sure, will try to add subtitles

[@naveenraj9977]

@@rajasdataengineering7585 Thanks,that will be great

[@vidhikumar1664]

Great explanation.

[@rajasdataengineering7585]

Glad it was helpful!

[@shekarsubramani9861]

Hi Raja, I was very much confused with the architecture, once I saw your video ,now its clear, Keep up the good work

[@rajasdataengineering7585]

Thanks Shekar!

[@AnandKumar-dc2bf]

nice pictorial representations bro keep gng

[@rajasdataengineering7585]

Thanks Anand

[@JindamSrilekha]

Well Explained

[@rajasdataengineering7585]

Thanks

[@vishalaaa1]

Excellent

[@rajasdataengineering7585]

Thank you! Cheers!

[@terrificmenace]

Excellent video 😀 thank you

[@alex45688]

good explanation

[@rajasdataengineering7585]

Thanks for liking

[@AnandKumar-dc2bf]

Nice explanation...

[@subhashyadav9262]

Very Nice

[@rajasdataengineering7585]

Thanks

[@tanushreenagar3116]

perfect video sir

[@rajasdataengineering7585]

Thank you!

[@shyamkumardhamode4475]

Soooo good explanation

[@rajasdataengineering7585]

Glad it was helpful!

[@arunshankar1987]

Exactly what am looking for. Please let me know where I can find the datasets to practice.

[@pridename2858]

Yes, this is master piece. Thanks

[@rajasdataengineering7585]

Welcome, Glad you like it!

[@pralgs628]

Awesome!! Could you please attach the PPT for Each Video.. Thanks

[@siddavatamvenugopalreddy9686]

Is this all tutorials related to spark only? Or it includes data bricks aswell? Please confirm

[@rajasdataengineering7585]

It's more on databricks which is internally using apache Spark

[@divyamariyameldo6495]

Thanks for the content!

[@rajasdataengineering7585]

My pleasure! Welcome

[@abinashsenapati1880]

It is really helpful. Thank you.. Where will I get the complete PPT of this playlist?

[@sharunkumar4806]

Hi Sir, Thank you for such an excellant material. I do not have access to Azure Data bricks at the moment. Can I still learn the complete playlist by installing pyspark on my local PC and pactising on Jupyter notebook ?

[@rajasdataengineering7585]

Thank you. Yes you can practice in your local installation

[@saravninja]

Started your videos!! All are great

[@rajasdataengineering7585]

Thank you Ninja

[@saravninja]

@@rajasdataengineering7585 I went through complete video second by second. Video has lot of clarity than any other KZbin channel. Keep up good work!! Have you experienced data skew issue, if yes can you point video or do video for us.

[@rajasdataengineering7585]

Thank you for your kind words. It gives a motivation to create more videos which can help genuine knowledge seekers like you. For data skew, have posted one video (though it does not cover advanced concepts) kzbin.info/www/bejne/e4LLnZevgbyDras

[@saravninja]

@@rajasdataengineering7585 thanks a lot again Raja!! Will go through it. I am looking for airflow training, I have sent mail to you, kindly respond.

[@krishnamurthy1243]

Hi Raja ,please do azure synapse analytics,eagerly waiting

[@rajasdataengineering7585]

Sure Krishna, will create videos on synapse analytics

[@mohammedmujahiduddin4715]

Thank you Raja for the detailed explanation. Do we have any video which is focusing on Worker Node and its details ? And as you were about to make a video regarding the memory management details, please also share that or the video title if already present. Thank you so much in advance!

[@rajasdataengineering7585]

Please watch videos kzbin.info/www/bejne/mYXNeaKhpN1of9U kzbin.info/www/bejne/d2mToGyNfL1-las

[@CoopmanGreg]

Great Video! 👍

[@rajasdataengineering7585]

Thanks 👍🏻

[@MichaelAdebayo-y4n]

Would watching these videos be enough to help pass the Databricks Certified Associate Developer for Apache Spark 3.0 - Scala exam?

[@rajasdataengineering7585]

Yes most of the concepts are covered in this channel.

[@ayanjit9196]

Sir can you please give us the links of the notebooks used in this series. This has helped me and a lot of other people. Giving this link would be even more helpful 🙏🙏🙏

[@100useful7]

Yes, please

[@manojru1]

I have installed Jupyter with Pyspark...where should I run my command to see the Spark job like you are showing on 38:21sec? or should I install some other IDE for that?

[@ArupSankarRoy]

Mistake 14:15 Partition Size should be 100 mb

[@vinothloganathan2623]

Hi Raja, One of the amazing explanation. I couldn't find these level of details in any of the source like - books, medium and other youtube. Amazing work !!. Could you share if there are any resource helped you for spark >

[@rajasdataengineering7585]

Thank you! I don't have any other resources. I summarised these concepts based on my working experience

[@sudippandit9855]

excellent explanation!!

[@rajasdataengineering7585]

Thank you Sudip

[@lanyofrancis1195]

Thanks for the videos and nice explanation, I have a question. Default partition size if 128 MB, so for 2 GB RDD file, no of partitions created will be 16. Here in your example are you changing the default size of partition to 10 MB instead of default 128 MB? Please correct if I am missing anything?

[@lakshmidvs3258]

Hi Raja, very nice explanation ... Are you taking classes

[@rajasdataengineering7585]

Hi Lakshmi, thanks for your comment! I'm not taking classes

[@lakshmidvs3258]

Hi Raja , Thank you so much for your quick reply . I want to write data bricks certification . It's show two exams 1.azure data bricks 2 . Databricks . For which certification I should go is there any difference in those two in getting job opportunities. I have experience as Data Engineer, now I want to do certification .

[@rajasdataengineering7585]

Both are same. You can go with databricks certified data engineering professional

[@oluakano6497]

hi, i am new to spark and your videos seem like a great resource to learn. i am wondering what is the best order to watch them? through the playlist pr just use the numbers like 1,2,3...

[@rajasdataengineering7585]

Hi, for all videos, I have given serial number. You can follow the order based on that serial number

[@jayaprakashm2849]

Nice info

[@kneelakanta8137]

can I have document for reference of this playlist

[@ElhamMirshekari]

Raja could you kindly make a video on these three functions and compare them: Join, Union, Concat

[@rajasdataengineering7585]

Hi Ellie, I have already created videos for join and union. Will make a video for concat as per your request. Join : kzbin.info/www/bejne/pHuqm3mDhaefisk Union: kzbin.info/www/bejne/fIW3fYB4gc6tjJo hope it helps you

[@ranganathhittanagi3315]

11:40 why is the 2GB file is divided into 200 partitions ? by default? isn't it file size 2GB should be divided by 128MB (block size) which is 16 partitions

[@RAVINDRAsap]

could you create CI CD for databricks please

[@codeslayer4713]

Hi Raja, I have a question here, in terms of partitions when we will be loading a file of 2gb the minpartitionbyte of 128 mb makes the initial partitions to be 16 with the logic 2*1024 / 128 right ? and the minpartition property has a number 200, but isnt it that if there is any shuffle operation then only 200 partitions will be there but not while reading

[@rajasdataengineering7585]

Yes, that's shuffling partition parameter which is not applicable for reading partition

[@codeslayer4713]

So you mentioned 200 in your example that's why my doubt arose

[@neelbanerjee7875]

Thank you ver much for such contents.. one request - Can you please make a video on real time executor number, core, memory allocation based on input data size like. 1. 1-5 gb 2. 5-15 gb 3. 15-25 gb 4. 25-50 gb 5. > 50gb = 1 tb

[@rajasdataengineering7585]

Sure Neel, will make a video on this requirement

[@mohitupadhayay1439]

@@rajasdataengineering7585 Waiting for this!

[@dipanjanpan1]

Can we create multiple executor node on a worker node?

[@rajasdataengineering7585]

Yes we can. Executor is logical division of computing resources

[@BlingKing321]

In order to store data in JVM memory we need to do serialization and deserialization. Why ?

[@shamsmalek]

Excellent job. Can you please provide me the data set and code? Or please give me the Git link to download the dataset and code for your tutorials. Thanks.

[@srikanthbachina7764]

HI Raj, Videos are Missing from 27 to 30 Could you Please Upload them.

[@rajasdataengineering7585]

Hi Srikanth, those 4 videos are related to Azure Synapse analytics. Its still available under all videos section

[@arupnaskar3818]

Hi Raja u r teaching is awesome .... really help .. sir, just wanted to know .. for "STAGING" here u mentioned about "Nodes" .. here "Nodes" means No of worker Nodes or Partitions ??

[@rajasdataengineering7585]

Thank you Arup. Node means no of worker nodes in the cluster

[@arupnaskar3818]

@@rajasdataengineering7585 thanks raja ..💐🌸

[@rudraganesh1507]

This is the masterpiece

[@rajasdataengineering7585]

Thanks Ajay

[@البداية-ذ1ذ]

please could you make videos in examples about pyspark in real projects

[@rajasdataengineering7585]

Sure, will make videos on real time projects

[@البداية-ذ1ذ]

Thanks alot

[@saifahmed7843]

Greetings, Is this sufficient for Databricks Certified Associate Developer exam? Appreciate any clarity.

[@rajasdataengineering7585]

Yes, I have covered almost all the topics. If you understand all the concepts I explained in this channel, that's more than sufficient

[@saifahmed7843]

@@rajasdataengineering7585 Thank you so much for the info.

[@rajasdataengineering7585]

Welcome

[@dinesh_tadepalli]

Are any prerequisites required to this pyspark series?

[@rajasdataengineering7585]

No, nothing needed. I have covered from basic

[@TelugupodcasterDT]

Thank you!

[@saikrishna1939]

Can you please guide me how to start your videos I mean the order I can see many playlists in the channel. I want to learn spark and data bricks

[@rajasdataengineering7585]

Sure bro, let me give serial number to my videos so that you can follow the structured learning list

[@saikrishna1939]

@@rajasdataengineering7585 yeah thanks, also please comment here that which playlists we shld follow for the order to learn spark and data bricks 🙂

[@rajasdataengineering7585]

Sure

[@rajasdataengineering7585]

@@saikrishna1939 The videos are given with serial number. You can follow with that sequence

[@saikrishna1939]

@@rajasdataengineering7585 yes but which playlist to follow as there are 7 playlists in the channel so it's Lil confusion. Say for example a playlist has 5 videos but when I open it I can see 17, 18, 19, 20 in the videos. In interview series it starts with 1 again

[@ElhamMirshekari]

Is the master node same as cluster manager? or they are two different concepts?

[@rajasdataengineering7585]

Master node is driver and different from cluster manager

[@ElhamMirshekari]

@@rajasdataengineering7585 Thanks for the prompt response . Master node = Driver

[@ParthKhambhayta-dj9te]

Sir you said when read CSV file it's divided in default 200 partitions but default size of block is 128MB so it should decide into 16 partition please let me know am I correct or not ?

[@anilmnt82]

I do see it as more detailed explanation on spark , but not really on Databricks, missing many Databricks features like Unity catalog, DBFS, Vaccum, Liquid clustering etc..

[@aashishmalhotra]

amazing

[@rajasdataengineering7585]

Thank you! Cheers!

[@morgann4276]

Great video raja!! Wanted to know how you have such in depth knowledge.. did you learn from spark docs ?

[@rajasdataengineering7585]

Thanks Morgan. Yes spark documents and working experience helped me to understand concepts

[@itsallinyourhead3593]

Hi Raja, How do we set the number of executors in azure databricks ? like in this example the worker node is divided into 4 executors. Thanks in advance!

[@rajasdataengineering7585]

Hi, number of executors can be controlled using spark config parameter "spark.executor.instances". Number of cores per executor can be set by spark.executor.cores. Hope it helps

[@itsallinyourhead3593]

@@rajasdataengineering7585 thank you, so these parameters have to set while cluster creation meaning are these parameters at cluster level or can be changed/set by developers during etl/data processing?

[@rajasdataengineering7585]

It can be set at cluster level using init scripts or at notebook level using syntax spark.config.set()

[@itsallinyourhead3593]

@@rajasdataengineering7585 thank you 🙏

[@rajasdataengineering7585]

Welcome

[@VipinYadav-ii1ow]

Just starting to learn spark and databrics. Is this resource is enough to crack entry level data engineering job?

[@rajasdataengineering7585]

Yes definitely, these videos are more than enough to crack entry level job

[@Arvind-sr6ze]

i need prepare for apache spark programming with databricks certification,will this videos help me?

[@rajasdataengineering7585]

Yes, it will help

[@Arun-uw1hy]

Hi, here the executor mean processor (CPU) , Right? Because each node may have multiple CPU, and also each CPU can have multiple cores. So each cores has a separate executor.

[@rajasdataengineering7585]

Executor means logical division of nodes. That means combination of processor + memory + network

[@mannykhan7752]

Yun number of worker nodes?? What is Yun or yum??

[@shivayogihiremath4785]

I'm following this channel from couple of days now. The content and way of explanation is awesome. Good job my friend. keep up the good work. wishing you all the very best. one small suggestion, if possible, please try to avoid the initial music (which is played at the beginning of the video) at times it is annoying. thank you!

[@rajasdataengineering7585]

Hi Shiv, thank you for your valuable comments. I already removed this initial music. May be it is still there for only few initial videos.