02. Databricks | PySpark: RDD, Dataframe and Dataset

[@chiranjivmansis1415]

The best Data Engineering course in youtube. Thanks a lot bro for your effort and that to free of cost. Really proud of you!

[@rajasdataengineering7585]

You are most welcome

[@reach2puneeths]

very informative, please come up with end to end projects using databricks

[@shivachaitanyachinna9819]

Thanks for providing indepth knowledge about these topics. Amazing.

[@rajasdataengineering7585]

Glad you like them! My pleasure!

[@amanpathak7507]

Hi, could you please provide the slides and notebooks, that would be really helpful for a quick revisions before interview

[@learningruchi]

Thank you for providing such detailed videos.

[@rajasdataengineering7585]

Glad you like them! Keep watching

[@bharatpogul6014]

Very nicely explained the concepts.

[@rajasdataengineering7585]

Glad you liked it! Thanks

[@ranjansrivastava9256]

As per your slide for the Differences among the RDD, Dataframe and Dataset- you mentioned the supported language for Dataframe is Java, Scala, Python and R. What about the SQL for these. Could you please clarify on this Raja. If possible.

[@rajasdataengineering7585]

Hi Ranjan, yes spark SQL is also supported by dataframe api

[@ourmind8677]

A doubt: As you said, ultimately spark converts dataframes into RDDs while processing. Then how the benefits like avoiding GC-process and others will eventually comes into play while using DFs instead of RDDs? I'm fairly new in this area. And thanks for this playlist.

[@rajasdataengineering7585]

GC is related to on heap memory, not related to dataframe or RDD.

[@pavanjavvadi9902]

So does it mean dataframes don’t run in heap memory ?

[@dineshdeshpande6197]

Hi Raja Sir, The contents are very good in this video and playlist. But not able to understand the sequence to follow as the numbers are missing in serial numbers you given. Also playlist has 65 videos but the serial numbers are above 100 also, can you pl help with sequencing of videos to go through the playlist.

[@maruthiraoyarapathineni2012]

Great work. 👍👏👏

[@rajasdataengineering7585]

Thank you! Cheers!

[@Ustaad_Phani]

Very informative

[@rajasdataengineering7585]

Glad it was helpful!

[@Abdullahkbc]

Hi, could you please activate the subtitles for this and other videos? these are really great sources, i don't wanna miss anything.

[@rajasdataengineering7585]

Hi Abdul, sure will activate the subtitles

[@Christy-du9jw]

@@rajasdataengineering7585 I would also appreciate the subtitles so I don't miss information

[@rkjunnu7224]

May I know the first video of the series?

[@Sandani_Aduri_Group]

Hi Raja, Your videos are very informative and interms of RDD/DataFrame/Dataset if some one which one is faster in execution what would be your answer?

[@rajasdataengineering7585]

Hi Sandani, good question. RDD is native api for Spark. So whatever we use dataset or dataframe, it would be internally converted to RDD. But rdd is quite outdated for programming nowadays. Dataframe is widely used across projects due to developer convenience. Would recommend to go with dataframe. Dataset has limitations with programming languages. For detailed information, please refer this video kzbin.info/www/bejne/nWW3Y2iVaa16g5I

[@ramangangwani9203]

sir can you please explain what is serialization

[@rajasdataengineering7585]

Sure, will create a video on this requirement

[@Abdullahkbc]

Hi Raja, could you please fix the order of the playlist? thanks in advance

[@rajasdataengineering7585]

Hi Abdullah, sure I will do it

[@gulsahtanay2341]

Thank you

[@rajasdataengineering7585]

You're welcome

[@harithad1757]

amazing

[@rajasdataengineering7585]

Thank you! Cheers!

[@kanstantsinhulevich4313]

dataset also has catalyst optimizations, but in slide it is just "optimization"

[@rajasdataengineering7585]

Yes dataset and spark SQL also uses catalyst optimizer. Optimization means catalyst optimizer. In the previous slide, mentioned that dataset consolidates best features from both rdd and dataframe

[@labib8aug]

Could you make a repo for all your videos.. Otherwise it is hard to follow you , thanks a lot Raja

[@meghagavade8672]

Best One

[@rajasdataengineering7585]

Thanks!

[@sorathiyasmit8602]

Your content is very good can you provide pdf of ppt

[@velaatechsolutions9738]

Super

[@premsaikarampudi3944]

RDD is not type safety right? they don't enforce datatype; This means that the type of the data in an RDD can change at runtime. This can lead to errors if the data is not properly checked.

[@rajasdataengineering7585]

Pls check spark official documentation instead of chatgpt to know the truth

[@gunar4831]

So pyspark uses dataframe and not dataset right?

[@rajasdataengineering7585]

Yes dataset is only available in scala and Java while dataframe is available with pyspark, R, scala, SQL

[@krishnamohan5950]

Can you please provide sequence number for your vedioes please

[@rajasdataengineering7585]

Sure Krishna, I will arrange the videos and create perfect playlist. Please allow me sometime for that.

[@krishnamohan5950]

Ru providing real time training raja ji

[@krishnamohan5950]

@@rajasdataengineering7585 sent email

[@rajasdataengineering7585]

Thanks, will respond asap

[@NikhilGosavi-go7be]

done

[@navjotsingh-hl1jg]

sir can you share pdf sir

[@aravind5310]

DataFrames are strong Type safety and RDD are not right. I think you need modify the slide.

[@rajasdataengineering7585]

No, dataframes are weak type safety, whereas rdd and datasets are strong type safety. For spark engine, dataframe is collection of rows (not individual columns) so it can't validate the column data type during compile time. So it is not strong type safety. Hope you understand. Pls refer spark documentation to know more about type safety

[@akash4517]

Dataframes are mutable .

[@rajasdataengineering7585]

No, dataframe is immutable

[@akash4517]

In Pyspark we can do this Df = Df . Select or any other transformation . Which will change its state ? Or am I considering mutability wrong ? .

[@rajasdataengineering7585]

Yes you can do df=df.select but it does not mean that dataframe is mutable. What happens internally is previous dataframe is dropped and another new df is created based on lazy evaluation, not the previous df is getting modified. Dataframe is always immutable

[@akash4517]

Ok thank you Raja for helping out . Got it .

[@akash4517]

Raja i am confused between two topics , optimize write and auto compact . I saw you had made video on optimize still confused .

[@GovardhanaReddy-kp6jt]

Raja Bro could you please provide your email id i need to learn This couse