How to Remove Duplicate Records in Azure Data factory | ADF Interview Questions & Answers

[@MaheshNuvvula-u8d]

Your videos are amazing. I learnt a lot from you. By mistake, you removed unique records too. It's wrong bro. If there are 6 records, 5 records are duplicate then one must be unique. But, in this video you are considering 6 records as duplicate. Can you please make another video and delete this video.

[@jayong2370]

To the point! Great thank you.

[@ASKUMBHAR]

Hi Sir, Thank you for giving the answer for removing duplicate records but at the end this answer is wrong. Removing duplicates means at the end It should keep one records of Amir shahzad in the unique list and The duplicate output file should Only contain one record of Amir shahzad. I we know IF Amir Shahzad has record comes twice that means one record is duplicate and one record is unique. In the practical scenario we will keep one record.

[@MegaBlertsie]

Great video my friend! Perhaps a next idea - do you know of a way to loop through a REST API call to return all the pages of the call if it has a next value in the API?

[@TechBrothersIT]

Thanks, I will think about this demo

[@yasmeenyash5644]

What if amir shahazad first name and last names are same for 3 rows how to remove those 2 duplicates and keep 1single record .please explain.thanks

[@vaibhavchaudhary1569]

You can use cte row_number as explained at the start. select just 1st row_number for each group will give you unique record.

[@vishaljhaveri7565]

In this video, I learnt that after performing the Aggregation step in which we create a column of COUNT (*), and if we want to split the columns based on this COUNT column, we conditionally split it where we write the Count > 1 contents saved in SINK 1 and unique content(not repeating) in SINK 2.

[@shanmugavadivelArts]

nice

[@sravankumar1767]

Nice explanation 👌

[@TechBrothersIT]

Thank you so much 🙂

[@sahilrana2659]

vdos are amazing,,why so much of fake accent?

[@TeluguMuchati]

I need a training from you ? Can you please respond to my msg