Create a Bank Account System with Kafka Streams - Stateful Processing

Kafka Streams is well prepared to handle stateful processing of streaming data. It has abstractions, operations and heavy lifting to help us in this difficult task.
In this tutorial we are going to build a Bank Account system to process live transactions and update the balance of each user.
---
Follow me:
Twitter: / progwithmati
LinkedIn: / matiasdsalerno
GitHub: github.com/Programming-with-M...
---
Sections:
00:00 - Intro
00:28 - Why stateful processing?
01:32 - State Stores in Kafka Streams
03-01 - KTable
03:32 - Stream/Table Duality
04:14 - Introducing our tutorial
06:01 - Browsing the source code
09:25 - Implementing the Topology
10:39 - Topology: Aggregate Operation
17:01 - Implementing the Process Transaction method
22:52 - Run application
25:50 - Outro, next episode...
---
Links:
GitHub repository: github.com/Programming-with-M...
Previous tutorials can be found in the GitHub page: programming-with-mati.github....

[@TriggerrSupernova]

As a beginner to Kafka streams I found the explanation really clear. Great explanation Mati Thanks.

[@twodots1990]

Awesome explanation 👏

[@pedrocorreia5857]

Thank you for the information, there really isn't much quality info about streams online so once again thank you and keep up the good work!!!

[@programmingwithmati]

Thanks Pedro! I'm glad it was helpful to you

[@rupeshjha02]

Good explanation and well organized videos. Looking forward for more such videos.

[@programmingwithmati]

Thanks for the feedback! Really appreciated! More videos are on the way!

[@ayyoubtelmoudy574]

I miss the words to describe you ,so thank you so much ❤

[@programmingwithmati]

You are welcome! Thanks for the kind comment!

[@thesadanand6599]

Great Work, There are very few doing playlists on kafka-streams end-to-end. would have been great, if you could have picked some easier short examples to help out begineers.

[@programmingwithmati]

Thanks for your kind comments and your feedback!

[@iliapetrov1743]

Hi Mati! Thanks once again for your videos! Could you please create any example of using Kafka as an interaction system among micro services if possible?)

[@programmingwithmati]

I will! Currently I’m preparing a video on how to scale a stateful application, which is taking me longer than I expected! Thanks for the feedback!

[@eybertalexis]

Where is the console-consumer script file?

[@pritamshirbhate6801]

nice explanation. how can we store the bank balances into the redis db using state store, so that we can fetch it whenever required?

[@programmingwithmati]

Hi Pritam, thanks for the comment! I haven’t seen an implementation of stateStore with Redis. So, as far as I know, you’ll have to create your own implementation, which you can absolutely do! You can probably look in the internet if someone created an implementation already.

[@wangfred]

Hi Mati, great content! I really appreciated your work. I have three questions: 1. noticed in BankBalanceTopology when filter by state==rejected, looks like the stream is's split to two branches, can you explain why we don't nee split().branch() here? 2. when you declare the rejected stream, how does it know which point of the stream it plugs in? this case, look like the order to declare the balance topic and rejected topic doesn't matter. 3. in the second test case, I understand the first balance is 0 and second one is 3000. But for the third one, it should be 3500, why it is 500?

[@marquardtsnell7916]

Oh boy, a test! I think I got this: 1. split().branch() would create an either/or condition. We still want to send the rejected transactions to the "bank-balances" topic, it's used for the **latestTransaction** field. So, we always send to "bank-balances" topic, but if it's rejected, we *also* send to "rejected-transactions" topic. 2. because of the first answer, you're correct. Once we have enough information to determine the value of the state (after deserialization), we can filter and send in any order. Think of sending to the "bank-balances" topic and filter/send to the "rejected-transactions" as parallel operations. You can do them in either order. 3. Notice the balanceIDs. In the first (-500) and third (+500) transactions, the balance Id is 1. In the second transaction (+3000) the balance Id is 2. The balance Ids are like account numbers, that's how we keep everyone's accounts separate. The transition happens in BankBalance, the BankBalance.id = bankTransaction.balanceId, then in the topology, the transactions are .groupByKey() and aggregated *by those keys*. Keeping balanceId#2's account safely away from balanceId#1!