08. Databricks | Pyspark: Add, Rename and Drop Columns

[@sowjanyagvs7780]

What is Lit() : whenever we want to add a constant literal value to entire data frame, then we go with LIT(). we can also add these values only to certain records using when and otherwise. Eg: EMPDF = df.withcolname("Bonus",when(df.sal>50k, lit(sal*10)).otherwise(lit(sal*20))....Thanks for the amazing session Raj sir

[@anthony281190]

Excelent, From Dominican REpublic

[@rajasdataengineering7585]

Thank you

[@Jaipreksha]

Thank you Raja. Very nice video

[@rajasdataengineering7585]

Thanks and welcome, Siva!

[@gulsahtanay2341]

Great content

[@rajasdataengineering7585]

Thanks!

[@kartikeshsaurkar4353]

Thank you Raja. Keep it up. Please zoom a little bit or increase the font size in lab sessions to reduce the eye strain

[@rajasdataengineering7585]

Sure, I will increase the font size

[@bonasiuday]

Very nice explanation 👏

[@sravankumar1767]

superb bro.........

[@tusharagarwal3553]

Where can we get these notebook..please share link for all the notebook.. so that we can revise

[@HPan-y2v]

Two things to add over here : 1. If we have a column name that doesn't exist in the spark dataframe, then drop will just ignore it and not throw any error: df.drop("XYZ"). printSchema() won't throw any error in spark 3.0 2. If you want to drop multiple columns , columns should be passed ONLY as strings : df.drop('firstname', 'lastname'). printSchema() If we will pass them as column then there would be an error. The below code would run into an error: df.drop(col('firstname'), col('lastname')). printSchema()

[@supriyakoura7755]

not for me emp_DF.drop(col('gender'),col('salary')).show()

[@Ravishekar-qd4fc]

I have to modify the column values of existing record like location is India that alter to AUS. How to handle

[@unbeatableinsaan4926]

Please make whole basics playlist by topic wise

[@rajasdataengineering7585]

Sure, will do

[@lakshminarayana3168]

hi Raja can you share the list of content uploaded to this playlist

[@mohammadumair7521]

will you upload all the notebook in you google drive and provide us the link of the notebook it will be the greater help of all I guess.

[@agamgupta-ed9ye]

Hi Raja, can you kindly share data sets with us so that we can practice also

[@srinivasarao416]

Hi Sir, I am kindly requesting could you please share notebooks

[@pavanrocks9960]

Hi raja can you kindly share data sets and codes for better practice

[@GurudasKundukar]

how to add identity coumn

[@rajasdataengineering7585]

Hi Guru, will post a video soon on identify column

[@patnaik476]

Very nice playlist. I have only one suggestion. Please don't use the ear deafening music in the start. Its very very loud and annoys everyone around.

[@rajasdataengineering7585]

Agreed. I have already changed this music in later videos. Thanks for your suggestion