Developing Real-Time Data Pipelines with Apache Kafka

Speaker: Joe Stein
Big Data Track
Slides: www.slideshare.net/SpringCentr...
Developing Real-Time Data Pipelines with Apache Kafka kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

[@regul4rjohn]

Exactly what I needed and perfectly laid out, thanks a lot.

[@gurkanwalsingh5680]

Thanks! A really great video for diving into Kafka.

[@SPeri6]

Best Kafka talk

[@khatuntsovmikhail6223]

ever seen

[@growlingchaos]

Great talk! Thanks!

[@sanke7982]

Great one.... in depth...

[@Praveenmech90]

Awesome talk!

[@InathiM]

Thanks for this. This is great !

[@daniel1988evil]

Fantastic talk!

[@srikanthjnr]

Excellent presentation dude!

[@christopherhahn6166]

Thanks Joe!

:) Very good talk!

[@shakthiramalingam3178]

Great talk.

[@nikolozrb8080]

great job

[@nickdecoursin]

Thank you!

[@arunbm123]

nice talk...

[@HafidzJazuli]

Can anybody recommend me similar talk, but simpler and straight forward? Honestly, I am seeking similar mechanism but more complex like ansible.

[@antonalekseyev9651]

What about monitoring kafka cluster? How to check whether cluster is alive?

[@sagarrout007]

You can use Yahoo Kafka Manager

[@vermajiutube]

shouldn't the leader election be based on offset number ?? (Broker which has the largest offset should be the leader)

[@parthosarathimaxotek]

My thoughts exactly! Maybe they favour uptime over message durability in such a scenario.

[@gokhan1342]

Where can i find slides? Can someone upload it?

[@srikanthjnr]

Check the description

[@MeghaGupta18]

this is really very knowledgeable talk. now is there no role of zookeeper ?

[@karthikgolagani6844]

those roles will be handled by controllers

[@SanjayRoy-gm8ej]

Kafka on a typical standalone implementation would require Zookeeper for coordination but in case of Mesos (i have never used it), is taken care of the scheduler and executor i guess..

[@madhukargovind]

You explained theritically But, not explained technically in Dev, Deployment and in productijn. Thanks

[@sailendramishra1560]

C

[@adityagary50]

13:30 initial flow

[@maheshdev5877]

Atleast he can develop a simple hello world application for better understanding It is very disappointing

[@nguyenhohai88]

Yo Spring should change the way you transfer knowledge to people. The presentation is very bad because lack of clear visualization, being interrupted by unclear questioning and answering. The video is way too long for small knowledge ...