22. distinct() & dropDuplicates() in PySpark | Azure Databricks

[@mallikarjun_reddy]

Wonderful content, More power to you. Please continue the next videos. I am not pushing, just checking to plan my things accordingly. Actually your videos are more better than paid content.

[@Bhaweshgupta]

I am following your pyspark series it is wonderful. Thanks for making it. Keep them coming I am learning a lot

[@WafaStudies]

Thank you 😊

[@akash4517]

Hi Maheer , Just one suggestion Can you Skip the notebook and dataframe creation part , its actually little redundant to see in multiple video and instead add scenario with function not just running the function. because for understanding how the functions work its easier we have multilple website . This were my suggestions it may vary for others but take it in a good way , you are doing very great job . i have seen most of your videos .

[@WafaStudies]

Thank you for suggesting. I will try to copy paste the dataframe creation part in upcoming videos. I usually try write code everytime so that people by watching multiple times get the code easily without practice. My main intension of PySpark playlist is to explain concepts with small examples. There is another playlist which I am building parallel called PySpark realtime scenarios there i try to explain big examples with detailed scenarios. Will take up ur suggestions as well. Thank you ☺️

[@anthony281190]

Thanks Bro, I resolved a problem with this video

[@shriyanshasthana2499]

Hello, I have one query In the above scenario, if we pass gender and salary in drop duplicates, shouldn't it show all the rows where the combination of salary and gender is unique?

[@samyboulos4642]

Great playlist. I suggest you don't create a DataFrame from scratch every time but use a pre-existing one in the interest of saving time.

[@polakigowtam183]

Thanks Maheer. Good vedio

[@WafaStudies]

Welcome 😇

[@akarshsharma7849]

This series completion time will be ? I am not pushing actually I am planning my things because you are explaining better than paid tutors ❤ thanks in advance 😊

[@WafaStudies]

I am trying to do daily atleast one video at this moment in PySpark playlist. U can follow and stay connected by subscribing to chennal. Hopefully i will try to cover as many videos as possible on this course 🙂

[@subhanishaik8163]

Hi bhai could you please make a video related to datetime functionalities like to_date(),timestamp and changing string to date

[@WafaStudies]

Sure will do. Thank you 😊

[@sanjeevpandey2753]

Maheer, the drop duplicate part has not given result as per sql and it has dropped both wafa from the df, may be a disclywould help freshers

[@manu77564]

Thank you bhaii

[@WafaStudies]

Welcome 😇

[@chidellasrinivas]

Hi Maheer, in this video at 6.58 why it dropped other Male Records and given Maheer Record.

[@sanjeevpandey2753]

Because it is just dropping duplicate from the gender column and not taking in account other columns, nice that you noted

[@harithad1757]

Please cover delta lake and tables concept

[@vutv5742]

Completed