02. Databricks | PySpark: RDD, Dataframe and Dataset

#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@reach2puneeths]

very informative, please come up with end to end projects using databricks

[@amanpathak7507]

Hi, could you please provide the slides and notebooks, that would be really helpful for a quick revisions before interview

[@user-le3ix8vg7j]

Thank you for providing such detailed videos.

[@rajasdataengineering7585]

Glad you like them! Keep watching

[@velaatechsolutions9738]

Super

[@maruthiraoyarapathineni2012]

Great work. 👍👏👏

[@rajasdataengineering7585]

Thank you! Cheers!

[@dineshdeshpande6197]

Hi Raja Sir, The contents are very good in this video and playlist. But not able to understand the sequence to follow as the numbers are missing in serial numbers you given. Also playlist has 65 videos but the serial numbers are above 100 also, can you pl help with sequencing of videos to go through the playlist.

[@Sandani_Aduri_Group]

Hi Raja, Your videos are very informative and interms of RDD/DataFrame/Dataset if some one which one is faster in execution what would be your answer?

[@rajasdataengineering7585]

Hi Sandani, good question. RDD is native api for Spark. So whatever we use dataset or dataframe, it would be internally converted to RDD. But rdd is quite outdated for programming nowadays. Dataframe is widely used across projects due to developer convenience. Would recommend to go with dataframe. Dataset has limitations with programming languages. For detailed information, please refer this video kzbin.info/www/bejne/nWW3Y2iVaa16g5I

[@labib8aug]

Could you make a repo for all your videos.. Otherwise it is hard to follow you , thanks a lot Raja

[@meghagavade8672]

Best One

[@rajasdataengineering7585]

Thanks!

[@ranjansrivastava9256]

As per your slide for the Differences among the RDD, Dataframe and Dataset- you mentioned the supported language for Dataframe is Java, Scala, Python and R. What about the SQL for these. Could you please clarify on this Raja. If possible.

[@rajasdataengineering7585]

Hi Ranjan, yes spark SQL is also supported by dataframe api

[@ourmind8677]

A doubt: As you said, ultimately spark converts dataframes into RDDs while processing. Then how the benefits like avoiding GC-process and others will eventually comes into play while using DFs instead of RDDs? I'm fairly new in this area. And thanks for this playlist.

[@rajasdataengineering7585]

GC is related to on heap memory, not related to dataframe or RDD.

[@pavanjavvadi9902]

So does it mean dataframes don’t run in heap memory ?

[@harithad1757]

amazing

[@rajasdataengineering7585]

Thank you! Cheers!

[@simachala]

can we have the github link for these PPT and code.

[@rkjunnu7224]

May I know the first video of the series?

[@gulsahtanay2341]

Thank you

[@rajasdataengineering7585]

You're welcome

[@Abdullahkbc]

Hi Raja, could you please fix the order of the playlist? thanks in advance

[@rajasdataengineering7585]

Hi Abdullah, sure I will do it

[@ramangangwani9203]

sir can you please explain what is serialization

[@rajasdataengineering7585]

Sure, will create a video on this requirement

[@kanstantsinhulevich4313]

dataset also has catalyst optimizations, but in slide it is just "optimization"

[@rajasdataengineering7585]

Yes dataset and spark SQL also uses catalyst optimizer. Optimization means catalyst optimizer. In the previous slide, mentioned that dataset consolidates best features from both rdd and dataframe

[@premsaikarampudi3944]

RDD is not type safety right? they don't enforce datatype; This means that the type of the data in an RDD can change at runtime. This can lead to errors if the data is not properly checked.

[@ranaumershamshad]

I checked this with ChatGPT. It says that RDDs in Spark offer flexibility and can handle various data types and do not provide strong type safety by default. To ensure type safety in RDD-based Spark applications, you should use best practices, perform explicit type checks and conversions, and consider higher-level abstractions like DataFrames and Datasets for structured data processing tasks.

[@rajasdataengineering7585]

Pls check spark official documentation instead of chatgpt to know the truth

[@krishnamohan5950]

Can you please provide sequence number for your vedioes please

[@rajasdataengineering7585]

Sure Krishna, I will arrange the videos and create perfect playlist. Please allow me sometime for that.

[@krishnamohan5950]

Ru providing real time training raja ji

[@krishnamohan5950]

@@rajasdataengineering7585 sent email

[@rajasdataengineering7585]

Thanks, will respond asap

[@navjotsingh-hl1jg]

sir can you share pdf sir

[@gunar4831]

So pyspark uses dataframe and not dataset right?

[@rajasdataengineering7585]

Yes dataset is only available in scala and Java while dataframe is available with pyspark, R, scala, SQL

[@aravind5310]

DataFrames are strong Type safety and RDD are not right. I think you need modify the slide.

[@rajasdataengineering7585]

No, dataframes are weak type safety, whereas rdd and datasets are strong type safety. For spark engine, dataframe is collection of rows (not individual columns) so it can't validate the column data type during compile time. So it is not strong type safety. Hope you understand. Pls refer spark documentation to know more about type safety

[@akash4517]

Dataframes are mutable .

[@rajasdataengineering7585]

No, dataframe is immutable

[@akash4517]

In Pyspark we can do this Df = Df . Select or any other transformation . Which will change its state ? Or am I considering mutability wrong ? .

[@rajasdataengineering7585]

Yes you can do df=df.select but it does not mean that dataframe is mutable. What happens internally is previous dataframe is dropped and another new df is created based on lazy evaluation, not the previous df is getting modified. Dataframe is always immutable

[@akash4517]

Ok thank you Raja for helping out . Got it .

[@akash4517]

Raja i am confused between two topics , optimize write and auto compact . I saw you had made video on optimize still confused .

[@Abdullahkbc]

Hi, could you please activate the subtitles for this and other videos? these are really great sources, i don't wanna miss anything.

[@rajasdataengineering7585]

Hi Abdul, sure will activate the subtitles

[@Christy-du9jw]

@@rajasdataengineering7585 I would also appreciate the subtitles so I don't miss information

[@GovardhanaReddy-kp6jt]

Raja Bro could you please provide your email id i need to learn This couse