Handling changes in Data in Bigdata world |Change data capture| SCD Types

[@jaskirank4137]

Very detailed and understandable information. Thanks

[@BigDataThoughts]

Thanks

[@sriadityab4794]

Very Well explained !!! Thank you

[@BigDataThoughts]

Thanks Aditya

[@pravinmahindrakar6144]

Well explained!

[@muzakiruddin266]

Very informative

[@kathirvelu3806]

When you said, over write - how the deleted records will be taken care... do you mean erase everything what you have and re-load?

[@himanshgautam]

Good information. Could you also provide which method you commonly use for capturing changing data from the source? I know of services like AWS DMS and golden gate for oracle. Is there any other method that we can use?

[@BigDataThoughts]

We need to write queries to track the change based on what we are handling I/U/D as explained in the video. In databricks there is a merge into command that can be used to do the same.

[@itriggerpeople]

Very informative ! As a ETL Tester, It helped clear my concept. Thanks Mam

[@ashishambre1008]

Can we implement scd in apache pyspark(not on databricks)?

[@BigDataThoughts]

SCD is a concept we can implement in any language we want

[@ashishambre1008]

I believe pyspark doesn’t support update and delete, so not sure how to implement and there isn’t much content on this topic elsewhere. Can you please create an example of this, I’m looking for scd type2 from a long time using pyspark but didn’t get any good answer

[@ASHISH517098]

​@@ashishambre1008did you find a way to implement scd in pyspark?

[@prabhatsingh7391]

@@ASHISH517098 yes SCD1 and SCD2 can be implement through pyspark.

[@sindhuchowdary572]

lets say there is no change in records for the next day.. then.. does the data gets overwrite again?? with same records..??

[@BigDataThoughts]

No we are only taking the new differential data when we do CDC

[@shivsuthar2291]

how will we know of deleted records as it does not come with incremental load

[@BigDataThoughts]

Only way to know about deleted records is if we get full load and we can do a diff. Or in case of incremental the upstream explicitly sends that information to us.