Change Data Capture (CDC) Explained (with examples)

[@swyxTV]

good topic choice and visuals! subscribed, keep it up

[@irtizahafiz]

Thank you so much! Hope you enjoy the future videos too. Let me know if you have any feedback.

[@nadavge]

Thanks, you kept it simple and easy to understand!

[@muhammadkaiser3544]

Thank you! This was very helpful.

[@irtizahafiz]

Thank you! I will start posting again soon, so please let me know what type of content interests you the most.

[@dataisfun4964]

Beautiful, thanks.

[@irtizahafiz]

Thank you! I will start posting again soon, so please let me know what type of content interests you the most.

[@Daily_rand_memes]

thank you for this video! really informative!

[@irtizahafiz]

Glad it was helpful!

[@lesterlino3316]

Great explanation, thanks!!

[@irtizahafiz]

Glad you enjoyed it!

[@achamac-donald9229]

Great explanation

[@irtizahafiz]

Glad you think so!

[@rajaramau6370]

nice explanation . Thank you :)

[@irtizahafiz]

You are welcome!

[@amlord68]

where is the code example??

[@nandkarthik]

What are some of the tools that provide CDC? Does databases provide it? or Are there any generic services?

[@irtizahafiz]

Most DBs provides some mechanism. You will need to configure it though, and usually provide some type of message broker.

[@souravpakhira]

how to detect change in database schema like rename of table name or adding new column?

[@irtizahafiz]

That's a good point. TBH, I am not 100% sure. I believe, you might have to update the connector, and then refresh the existing data back into Kafka.

[@souravpakhira]

@@irtizahafiz nvm I have already found the solution and have implemented it

[@kartech11]

Lets say I have a order booking system that has order and order details table. Now , one order details has changed. I want to send a complete order event that comprises of order and order details to kafka so that it can be consumed and stored in a time series database as a complete order model. Where exactly will the order details be fetched , because CDC will only tell me order details has changed.

[@irtizahafiz]

Hi! Thank you for asking. So this is a very good use case of Kafka Streams. Let's say you have a CDC stream for `order` and one CDC stream for `order_details`. The two of these should be in two different kafka topics. Using Kafka Stream or KSQL, you can join the two streams whenever either changes. Do the join based on the `order_id`. Check this out: supergloo.com/kafka-streams/kafka-streams-joins-examples/

[@kartech11]

@@irtizahafiz Thank you so much for the explanation. I went through the kstreams join example.Lets say my kafka topics store only last 7 days worth of data. Now lets say 20 days later the order details changed so it was sent as an event to order details event topic. When I use kstreams to join with order, in the kafka topic to store order events, it wont find the order because its cleared out. So how is this handled in the above case?

[@frankdeng8]

How does the db send messages to kafka ?

[@irtizahafiz]

Hi! So there is usually a middle man between the DB and Kafka, something called a Connector. Debezium is a good example of that. What the connector does is read from the database's log files and writes to Kafka. Most databases (if not all) has some kind of a log where it records every DB operation. You can replay all the changes by reading this file. For Postgres you have WAL (write ahead log) and for MySQL you have other bin logs. So the connector reads from this log file and writes the changes to Kafka for every change you make to your data.

[@jamieridenour1275]

so this is transactional replication but for each individual table?

[@irtizahafiz]

I am not sure I understand what you mean by transactional replication. And yes, you can do it for individual tables.

[@andynelson2340]

nice explanation

[@irtizahafiz]

Thank you! Glad you found it helpful : )

[@mariofredrick1501]

how about upsert operation? is it supported by debezium?

[@irtizahafiz]

I believe it is.

[@nguyenngothuong]

thank

[@nr798yna]

Hi, its good explanation!. Could you make a video of how Microsoft SQL Server based CDC pushes messages to kafka ? I mean the implementation details! Thank You !

[@irtizahafiz]

Hi! I am not really familiar with Microsoft SQL Server, and currently its not in my plans :(

[@dndhndn]

is it safe to assume that CDC is just streaming concept of replicating & updating data between data sources?

[@irtizahafiz]

Yup! That's a really nice way to put it.

[@joaopedrom6337]

Deus abençoe pelo tradutor automático do youtube

[@yossra-elhaddad00]

Thanks for this simple great explanation

[@irtizahafiz]

Glad it was helpful!

[@khushaltrivedi9829]

is it near to realtime? if you have master db as rds where write will happen and u would want search as Elastic search but we need to stream data real time will this be real time?

[@irtizahafiz]

Depends on "how" real time your application needs to be. If you are feeding the CDC data into ES, I believe you will need to re-index which will take time. Personally, I haven't used that pipeline before, so I don't have too much context.

[@GabrielFerreira-is9ly]

Perfeito! gostaria de exemplos de uso em código como Node.js, Python

[@hp50537]

like I want to connect mysql to bigquery using pubsub how?

[@irtizahafiz]

There should be a Kafka connector you can utilize. I know Debezium has a few of them, but Google might also offer it as a service. One option might be to use GCP's MySQL equivalent, if you want native integration with BigQuery.