Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt

Have you ever thought that you needed to be a programmer to do stream processing and build streaming data pipelines? Think again! Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub-sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with RDBMS, NoSQL, and object stores is simple with Kafka Connect, which is part of Apache Kafka. KSQL is a SQL streaming engine for Apache Kafka, and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.
In this talk we’ll explain the architectural reasoning for Apache Kafka and the benefits of real-time integration, and we’ll build a streaming data pipeline using nothing but our bare hands, Kafka Connect, and KSQL.
Gasp as we filter events in real time! Be amazed at how we can enrich streams of data with data from RDBMS! Be astonished at the power of streaming aggregates for anomaly detection!

[@Ennem0_o]

Probably one of the best talk I've seen on Kafka capabilities. If you don't agree with me, please no thumb down, simply provide more relevant links 😉

[@TheRealKwena]

I love this! thank you, clear and simple.

[@rahult883]

Extremely helpful and to the point. Thank you so much!

[@rmoff]

Demo code: github.com/confluentinc/demo-scene/blob/master/build-a-streaming-pipeline/ Slides: rmoff.dev/ljc-kafka-01

[@olehprotsenko3561]

Thanks for the demo, would be great to have an example with KSQL (table) aggregation group by 2 or more fields as a key.

[@jishnuanchery]

great talk.

[@ranjithpals]

Very helpful in understanding some basics of ksql, streams, tables, topics and how are they are interconnected.

[@leoxiaoyanqu]

Great talk, and the pointer was pretty cool

[@alokkumartiwari6482]

Great demo, Extremely useful..

[@mahadyh]

great demo :)

[@2007selvam]

Really very good presentation Robin

[@Eagle-eyed978]

Excellent Talk

[@CarloL525]

Excellent demo

[@DevWonYoung]

awesome KSQL!! awesome Robin!!

[@krishnakummar16]

awesome demo

[@rajeshpednekar9317]

Good presentation for KSQL insights

[@gurumhase2198]

Very impressive session

[@harjeetkajal]

Great explanation !

[@darlingtonmatongo9436]

very very good presentation, thanks very much

[@soumikdutta77]

Such an amazing gateway introduced to us which talks about Streaming ETL. Thank you Mam . Hope to talk to you one day !

[@azurpazur]

Very nice

[@namanbhayani1016]

Awesome!

[@sogab2bab]

Is there any tutorial document for a JDBC-Connector ??

[@rmoff]

Yes, check out rmoff.dev/youtube for several tutorials using this connector

[@murali.m8026]

Which one is best for developers career, kafka or node js

[@jonnytheponny5753]

actually at least both: they are completely orthogonal skills. node a programme language and kafka an architectural / data persistence pattern. In a real world you would always need both.

[@krishnakummar16]

for developers, solve problems, write algorithms. kafka is just a tool

[@illiakailli]

17:22 LETS use CAPS FOR STREAM NAMES and TOPICS but not ALWAYS

[@danish162]

all those late comers, so distracting