02. Databricks | PySpark: RDD, Dataframe and Dataset

#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@chiranjivmansis1415]

The best Data Engineering course in youtube. Thanks a lot bro for your effort and that to free of cost. Really proud of you!

[@rajasdataengineering7585]

You are most welcome

[@reach2puneeths]

very informative, please come up with end to end projects using databricks

[@shivachaitanyachinna9819]

Thanks for providing indepth knowledge about these topics. Amazing.

[@rajasdataengineering7585]

Glad you like them! My pleasure!

[@maruthiraoyarapathineni2012]

Great work. 👍👏👏

[@rajasdataengineering7585]

Thank you! Cheers!

[@bharatpogul6014]

Very nicely explained the concepts.

[@rajasdataengineering7585]

Glad you liked it! Thanks

[@learningruchi]

Thank you for providing such detailed videos.

[@rajasdataengineering7585]

Glad you like them! Keep watching

[@Ustaad_Phani]

Very informative

[@rajasdataengineering7585]

Glad it was helpful!

[@meghagavade8672]

Best One

[@rajasdataengineering7585]

Thanks!

[@amanpathak7507]

Hi, could you please provide the slides and notebooks, that would be really helpful for a quick revisions before interview

[@harithad1757]

amazing

[@rajasdataengineering7585]

Thank you! Cheers!

[@cer_oz]

Hi, could you please activate the subtitles for this and other videos? these are really great sources, i don't wanna miss anything.

[@rajasdataengineering7585]

Hi Abdul, sure will activate the subtitles

[@Christy-du9jw]

@@rajasdataengineering7585 I would also appreciate the subtitles so I don't miss information

[@velaatechsolutions9738]

Super

[@gulsahtanay2341]

Thank you

[@rajasdataengineering7585]

You're welcome

[@sorathiyasmit8602]

Your content is very good can you provide pdf of ppt

[@kanstantsinhulevich4313]

dataset also has catalyst optimizations, but in slide it is just "optimization"

[@rajasdataengineering7585]

Yes dataset and spark SQL also uses catalyst optimizer. Optimization means catalyst optimizer. In the previous slide, mentioned that dataset consolidates best features from both rdd and dataframe

[@labib8aug]

Could you make a repo for all your videos.. Otherwise it is hard to follow you , thanks a lot Raja

[@Sandani_Aduri_Group]

Hi Raja, Your videos are very informative and interms of RDD/DataFrame/Dataset if some one which one is faster in execution what would be your answer?

[@rajasdataengineering7585]

Hi Sandani, good question. RDD is native api for Spark. So whatever we use dataset or dataframe, it would be internally converted to RDD. But rdd is quite outdated for programming nowadays. Dataframe is widely used across projects due to developer convenience. Would recommend to go with dataframe. Dataset has limitations with programming languages. For detailed information, please refer this video kzbin.info/www/bejne/nWW3Y2iVaa16g5I

[@ourmind8677]

A doubt: As you said, ultimately spark converts dataframes into RDDs while processing. Then how the benefits like avoiding GC-process and others will eventually comes into play while using DFs instead of RDDs? I'm fairly new in this area. And thanks for this playlist.

[@rajasdataengineering7585]

GC is related to on heap memory, not related to dataframe or RDD.

[@pavanjavvadi9902]

So does it mean dataframes don’t run in heap memory ?

[@dineshdeshpande6197]

Hi Raja Sir, The contents are very good in this video and playlist. But not able to understand the sequence to follow as the numbers are missing in serial numbers you given. Also playlist has 65 videos but the serial numbers are above 100 also, can you pl help with sequencing of videos to go through the playlist.

[@ranjansrivastava9256]

As per your slide for the Differences among the RDD, Dataframe and Dataset- you mentioned the supported language for Dataframe is Java, Scala, Python and R. What about the SQL for these. Could you please clarify on this Raja. If possible.

[@rajasdataengineering7585]

Hi Ranjan, yes spark SQL is also supported by dataframe api

[@NikhilGosavi-go7be]

done

[@ramangangwani9203]

sir can you please explain what is serialization

[@rajasdataengineering7585]

Sure, will create a video on this requirement

[@cer_oz]

Hi Raja, could you please fix the order of the playlist? thanks in advance

[@rajasdataengineering7585]

Hi Abdullah, sure I will do it

[@premsaikarampudi3944]

RDD is not type safety right? they don't enforce datatype; This means that the type of the data in an RDD can change at runtime. This can lead to errors if the data is not properly checked.

[@rajasdataengineering7585]

Pls check spark official documentation instead of chatgpt to know the truth

[@rkjunnu7224]

May I know the first video of the series?

[@gunar4831]

So pyspark uses dataframe and not dataset right?

[@rajasdataengineering7585]

Yes dataset is only available in scala and Java while dataframe is available with pyspark, R, scala, SQL

[@aravind5310]

DataFrames are strong Type safety and RDD are not right. I think you need modify the slide.

[@rajasdataengineering7585]

No, dataframes are weak type safety, whereas rdd and datasets are strong type safety. For spark engine, dataframe is collection of rows (not individual columns) so it can't validate the column data type during compile time. So it is not strong type safety. Hope you understand. Pls refer spark documentation to know more about type safety

[@krishnamohan5950]

Can you please provide sequence number for your vedioes please

[@rajasdataengineering7585]

Sure Krishna, I will arrange the videos and create perfect playlist. Please allow me sometime for that.

[@krishnamohan5950]

Ru providing real time training raja ji

[@krishnamohan5950]

@@rajasdataengineering7585 sent email

[@rajasdataengineering7585]

Thanks, will respond asap

[@navjotsingh-hl1jg]

sir can you share pdf sir

[@akash4517]

Dataframes are mutable .

[@rajasdataengineering7585]

No, dataframe is immutable

[@akash4517]

In Pyspark we can do this Df = Df . Select or any other transformation . Which will change its state ? Or am I considering mutability wrong ? .

[@rajasdataengineering7585]

Yes you can do df=df.select but it does not mean that dataframe is mutable. What happens internally is previous dataframe is dropped and another new df is created based on lazy evaluation, not the previous df is getting modified. Dataframe is always immutable

[@akash4517]

Ok thank you Raja for helping out . Got it .

[@akash4517]

Raja i am confused between two topics , optimize write and auto compact . I saw you had made video on optimize still confused .

[@GovardhanaReddy-kp6jt]

Raja Bro could you please provide your email id i need to learn This couse