03. Databricks | PySpark: Transformation and Action

#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@vydudraksharam5960]

Raja Sir, the way come well prepared with workbooks with example according to your explanation is too good. appreciate all your efforts and service to community.

[@rajasdataengineering7585]

It's my pleasure

[@JayJay-tk2gp]

This is a great work. The level of detailing is outstanding with an easy to understand teaching style. I really thank you for your time and effort you put into this video series.

[@rajasdataengineering7585]

Thank you very much!

[@shalabhkushwaha660]

Great work Sir. It was like i am learning since 100s of years. So simple to understand

[@rajasdataengineering7585]

Glad to hear that! Thanks

[@yuvrajchauhan9410]

Great video, deserves way more views and likes!

[@rajasdataengineering7585]

Much appreciated! Thank you

[@sachinchandanshiv7578]

Explained in very nice way. Thank you so much Sir🙏

[@rajasdataengineering7585]

Thank you Sachin

[@premamal1381]

Great Explanation Raja Sir. It would be really helpful if you can share the data files and code if possible so that we can do hands on along with the video.

[@aperez1969]

Great videos Raja! Thx a lot!!!!!

[@rajasdataengineering7585]

Thanks Alfonso!

[@kavithareddy5958]

Very Nice explanation Sir👌

[@rajasdataengineering7585]

Thank you! Keep watching

[@krishnachaithanyaudayagiri9806]

Hi Mr.Raja, It's a really good session. It's really good and helpful if you can share the files and scripts that you are using for the sessions.

[@user-cm7ei2dp9t]

Thank you soo much for this series of videos, this channel is so underrated kudos to ur hardwork, this will help me alot with my DE interview, Can u please make a video on the Azure DE project using ADF, ADLS, Databricks and pyspark and how does we deploy those production pipelines. THanks in advaance again

[@rajasdataengineering7585]

Thank you for your comment! I have already created videos on ADF+ databricks+ ADLS+ Pyspark. Pls refer videos 84 to 87

[@sayakbanerjee3572]

Great video sir !

[@rajasdataengineering7585]

Glad you enjoyed it

[@souravdey1227]

Great explanation

[@rajasdataengineering7585]

Thank you

[@o2udor277]

Very helpful, thank you.

[@rajasdataengineering7585]

Glad it was helpful! Thanks

[@hsdfunworld]

Excellent bro. Nice Explanation.

[@rajasdataengineering7585]

Thank you

[@pcchadra]

Videos are wonderful.Can you please share the noteboos and datasets please

[@savitajade8425]

Good explanation

[@rajasdataengineering7585]

Thanks, keep watching!

[@abhinavsingh706]

God bless you Raja !

[@rajasdataengineering7585]

Thank you Abhinav!

[@mallikarjun_reddy]

It would be better if you provide the source dataset and databricks notebook in description

[@rajasdataengineering7585]

Sure, will add the dataset in the description box going forward

[@gopichand3119]

explanation is superb please provide source dataset link for download

[@gulsahtanay2341]

Thank you

[@rajasdataengineering7585]

Welcome!

[@krishnakanthlenka2323]

It's nice session, it would be really great if you provide source data files that you are using.

[@sasanknimmagadda2010]

Yes I thought the same about source data, if it was in Azure blob/ Azure Datalake Gen2, then it would be much better. But this is a community version though. But if you could somehow show that process.

[@southernfans1499]

Hi bro, in new subscriber..videos are useful

[@rajasdataengineering7585]

Thanks bro

[@roymou5856]

Raja Sir, Sorry if i am asking any silly qs. pretty new to spark and databricks. example used for wide transformation is a count. but as you mentioned previously that count is an action. does that mean all actions needs shuffling of data and those are called wide transformation?thanks

[@alex45688]

it was great explanation can we get the data set for practice and for data bricks account did you made any specific changes in the account as I am new and not created an account yet

[@vinayveerabhadra7280]

Hi @Raja, You are executing Action command .count() in cell 12 itself, and on cell 14 you have executed one more action command .collect(). Please correct me if I am wrong. And in cell 14 there is a stage skipped for Job 23, is it because you execute a action command in cell 12

[@rajasdataengineering7585]

Hi Vinay, yes thats right

[@vinayveerabhadra7280]

@@rajasdataengineering7585 Perfect! I’d like to express my appreciation for your tremendous effort in creating this playlist and offering it for free. Thank you very much! 🙏

[@rajasdataengineering7585]

Thanks for your comment!

[@AnkurRanjan]

Hi, Would you mind sharing the dataset used in this video?

[@jaideep1222]

Do the intermediate dataframes df1 and df2 will get created in the spark memory during the execution or only the final df3 will be created ?

[@rajasdataengineering7585]

Yes df1 and df2 are also getting created

[@sureshkoduru8810]

Hi Raja, nice explanation in the video transformations list both Wide & Narrow transformation or only wide transformation

[@rajasdataengineering7585]

It contains both wide and narrow transformation

[@sreekanth6180]

it will be better if you provide the source dataset link so that we can download it for practice purpose

[@rajasdataengineering7585]

Thanks for letting me know this requirement. Sure, will provide the source data and code in description going forward

[@NikhilaMarripati]

where can i find the dataset

[@saravninja]

Really great explanation on given topic, question- in the demo, count is being used before collect action. Since count is also action then it should execute whole DAG logical plan till count instead execution happens only at collect action. Can you throw some light here.

[@rajasdataengineering7585]

Hi Sarav, good question. Group by count is aggregation in spark and comed under wide transformation, not action. So when we call collect function, it executes the DAG

[@stepup2me1]

@@rajasdataengineering7585 in your list of transformations and actions just before the demo, you have mentioned 'count' under Actionlist. But here in the demo you mentioned count() as transformation. Can you please clarify??

[@stepup2me1]

@@rajasdataengineering7585 example from your demo: if i assign like this --> df5 = df4.collect() would you call this as transformation ? since it creates another dataframe and doesnt show the output to the screen?

[@rajasdataengineering7585]

No it is action ..collect is for output so don't use it to assign to another dataframe

[@stepup2me1]

@@rajasdataengineering7585 so only those api's which show the outputs on the screen are called actions

[@gowthamsagarkurapati9388]

Hi Raja, Thanks for the video, to my understanding whenever a wide transformation is performed it should create a new stage, but in the DAG, i don't see any new stage created. please explain if my understanding is wrong.

[@rajasdataengineering7585]

Hi Gowtham, for wide transformation, new stage would be created for sure. There are certain scenarios where the same code was executed in same cluster then those stages would be skipped, as the data is already shuffled

[@pulkitgupta9263]

can anyone tell me from where I can download the csv files, to code along ?

[@pranaysawant7517]

Nice explaination but source data file is missing and how you install SPARK/PYSPARK? it is also missing

[@davidobembe5302]

Great videos Raja. Please can you procide your link to your data file so we can follow along in the demo?

[@rajasdataengineering7585]

Sure David, will do 👍🏻

[@davidobembe5302]

@@rajasdataengineering7585 will really appreciate. Your videos are really helpful and the explanations are clear. Big ups.

[@rajasdataengineering7585]

Thanks

[@adig626]

Hi brother, is it possible to upload the datasets also ? can you pls do it

[@sajidsarkar9574]

Could you provide the datasets or git repo?

[@user-zt3kc5we1f]

how to get dataset for practice?

[@telugucartoonchannel2916]

Can you please provide the links of data files , with out files we will understand but unable to practice. Please provide the data files to practice.

[@charchilgupta662]

Hi @rajasdataengineering7585 , can you please provide csv used in the course, course is excellent, and if you upload it is much help full.

[@suchismitasahu1810]

Hi Mr. Raja, can you please share any git repo for this?

[@kalyanpatel784]

Hi, Could you please provide the source dataset link to download ?

[@rajasdataengineering7585]

Hi Kalyan,I haven't yet setup repository. Will let you know once done

[@shardakhandve4896]

please provide csv links

[@sairahul740]

It would be great if you share the datasets used in the videos. without practise it's tough to learn

[@Ravikumar-op5qf]

Initial music is not good very inconvenient please remove

[@rajasdataengineering7585]

Sure, already removed in later videos

[@bhupeshaggarwal1862]

it will be better if you provide the source dataset link so that we can download it for practice purpose

[@rajasdataengineering7585]

Sure, will share the dataset