Handle Deduplication While Loading CSV File Into Snowflake | Ch-07 | Snowflake Data Loading Approach

[@ARUNKUMAR-qy2jt]

how to achieve this in s3 bucket? we won't load duplicates from s3 bucket to staging area

[@mayanknema3007]

Nice Video. Covered the in depth of duplication. Grear work..!!

[@DataEngineering]

Glad you liked it!

[@arindammitra3975]

Very nice presentation. Helped me a lot. Thank you.

[@DataEngineering]

Glad you linked the content... if you liked this.. you will also like this new series..end to end ETL using Snowpark kzbin.info/www/bejne/Z5umamuOhtx1kNk

[@siddhipatodia473]

This approach works if we set up snowpipe as well?

[@vvenakatesh]

i am loading same data and using different file name and same records, it will load and showing duplicate records in user table, how to avoid this kind of issue

[@sanjeev2012delhi]

Great learning plateform, really enjoyed😊😊

[@DataEngineering]

Glad you liked it

[@pragyakhare1762]

Great video! Any thoughts how can we check if data being loaded is not already in the table? and load only new data?

[@DataEngineering]

Would you like to check while loading new data set or you just want to check if data is loaded previously or not, for that you can check using copy history information schema. [Watch information schema chapter in my master playlist kzbin.info/www/bejne/oaK9g4ule8h-pZI]

[@mohammedvahid5099]

Excellent 👌 sir 🎉thnk u lot❤

[@DataEngineering]

Most welcome

[@hemaammu2337]

Hi sir, this helped me alot..thank you so much...one question- suppose if there are any server issues while loading the file and the file is loaded partially...if we try to reload the same file again , will it take the remaining unloaded data in file..?? So how can we handle ..? please help me with this

[@DataEngineering]

if the data is loaded from a file, partially or fully.. unless you change the file. (the MD5 hash, that is being tracked by snowflake).. snowflake will not pick it up... and if you fix the problem.. the duplicate records must be handled manually... there is no automated way..

[@Vishnugondi]

Is there any dump’s available for snow pro certification exam sir?

[@DataEngineering]

I am not sure about dump.. but you can go through this playlist to test your knowledge. Many folks cleared the exam by following this playlist along side snowflake documentation..hope this will also help. kzbin.info/aero/PLba2xJ7yxHB5X2CMe7qZZu-V4LxNE1HbF

[@Vishnugondi]

@@DataEngineering Thanks for your kind support sir..❤️

[@gujjarisravan1746]

Hi Sir this video is helped me a lot thank you for your effort could u upload unit testing for snowflake tables like we need to do unit test between before transformation table to after transformation table.

[@DataEngineering]

We will try

[@mayanknema3007]

Just one quick question - Under select distinct command - @~/ch07/csv/01_user_data.csvWhat is 't' here? You can refer this @ 5.10 Min.

[@DataEngineering]

t is an alias.. you can have any name like tbl..