Kafka Streams in Scala - a Comprehensive Tutorial

Written form:
blog.rockthejv...
In this video, we'll learn Kafka Streams in Scala, from scratch. We're going to go through all of its most important features on a realistic scenario:
how to set up a Kafka stream in Scala
how to transform data
how to join semi-static data with KTables
how to compose Kafka streams together
how to start a complete Kafka Streams application
Follow Rock the JVM on:
LinkedIn: / rockthejvm
Twitter: / rockthejvm
Blog: rockthejvm.com...
-------------------------------------------------------------------------
Home: rockthejvm.com
-------------------------------------------------------------------------

[@XArticSpartanX]

Absolute best programming channel I've found

[@ZoYaBoBo]

Absolutely agree !!!

[@camilongii]

Excellent!, maybe a part 2 with Avro

[@rockthejvm]

I'll have a full course for it

[@janga8717]

First things first: Thank You! Feels like self punishment - I am not very experienced in Scala at all, but I am trying to master KStreams by using Scala. Your tutorial helped a lot with that challenge! Just for give you something back: I had massive problems with SLF4J. This became very important when I produced broken code (which I did most of the time). Would be a good side-note for your blog, too add the dependency "org.slf4j" also to your sbt file. Otherwise some users (like I did) will run into issue like `SLF4J: Defaulting to no-operation (NOP) logger implementation` It might be obvious for more experienced Java / Scala developers than I am. But that hint would have save me some pain

[@rockthejvm]

Yes, logging is very important especially in distributed systems. Thanks for the note

[@sarfarazhussain6883]

Kafka Streams using Scala looks so clean and concise to me, at the same time easy to understand compared to Java syntax. Daniel, when can we expect a full course on "Kafka Streams using Scala"?

[@rockthejvm]

Glad you like it! ETA unknown

[@kotobotov]

awersome material, thanx for sharing

[@theanigos]

Here the boss comes

[@victorpre]

you are amazing

[@laiboonh]

Q: won't KTable run out of memory since its statically saved in broker?

[@rockthejvm]

I'll talk more subtleties in a full course on Kafka.

[@cgmds1973]

Hey Daniel, thanks for the article! When I try producing messages for discount-profiles-by-user I got this error: "WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {discount-profiles-by-user=LEADER_NOT_AVAILABLE" do you know how to fix it?

[@mbesida]

This lib is only applicable when there is a need to transform data inside Kafka? I mean if you need to enrich the data from some other system(via http call for example) can kafka-streams be used in this scenario? Also how conceptually it differs from ksqldb from confluent? Thanks.

[@zone66]

is kafka streams or alpakka to recommend? is one of them outdated?

[@rockthejvm]

Neither - they're used for different purposes.

[@victorpre]

hey Daniel, there is one typo in the blogpost on the last command: kafka-console-consumer.sh is actually trying to read from the wrong topic. In the code you write to "payed-orders" and on the kafka-consome-consumer you try to read from "paid-orders". all the best, cheers

[@rockthejvm]

fixed, thanks!

[@timofeygolovach1712]

Great video. Thnx! Is there ways to mix in messages of different types in the same topic? How will "Serde"s look like? 1 topic per 1 type does not sound good. Thnx

[@rockthejvm]

You can in theory push binary data which you can deserialize as you like. However, it might hurt your data organization.

[@timofeygolovach1712]

@@rockthejvm Can protobuf or Avro could be a solution for this?

[@zone66]

maybe kind of a wrong question to ask here, but if i use the confluent stack to set up a kafka cluster, it seems that i need to pay much much money. As far as i understand people mostly use the confluent kafka docker images, but it seems they only allow one node to be free without costing money? i would like to use kafka, but maybe my company is too small and has not enough money and resources for it. I mean, whats the point in using kafka if you only have one node, and the thing is therefore not really failure tolerant for a production system. Guess i stay with good old shared databases then to just safe and read everything from all microservices lol i hate reading through complicated pricing models as a developer. Can is set up apache kafka docker containers without confluent, or is that too much work?

[@rockthejvm]

There are many configs based on Docker online. You can use Kafka as a distributed message bus between your services, instead of relying on HTTP or other more cumbersome tools.

[@GershonBenYitzhak]

'Short'

[@conormcnuggets3556]

getting this error `There was an error converting the message profile1, Left(io.circe.ParsingFailure: expected json value got 'profil...'`

[@akashshah1925]

Create another implicit converter for key in scope. Maybe below the object's serde. implicit val stringSerde = Serdes.stringSerde You could also import Serdes._