Lessons learned form Kafka in production (Tim Berglund, Confluent)

[@sonunitjsr223]

One may directly jump to the discussion on production issues which starts at 30:24

[@tewikma231]

thanks for saving me 30 min :)

[@kiranprabhu420]

Thank you!

[@crosbykabir2418]

I know im asking the wrong place but does anybody know of a trick to log back into an Instagram account? I was stupid forgot my password. I love any assistance you can give me!

[@jonathankingsley3956]

@Crosby Kabir instablaster ;)

[@crosbykabir2418]

@Jonathan Kingsley I really appreciate your reply. I got to the site through google and Im in the hacking process now. Takes quite some time so I will reply here later when my account password hopefully is recovered.

[@nammals]

70% funny talks, 30% real content, start from 30:24

[@BayrangSafar]

Hearing this, I have also got the motivation that I too can be a speaker like him. 🙏

[@dontbescaredhomie3137]

Nice demo from Gavin Belson!

[@pradeep422]

wow its true!!!!!!!!

[@mikeskali1]

he is real! :)

[@ChodeOReilly]

Lol

[@brutusmaximumus]

lol

[@asganaway]

🤣

[@tekal85]

I watched the video to get some lessons learned in production, but more than 50% of it is Kafka basics terminology and marketing.

[@ankitlakum1]

live simple life is actually a solution ..👍

[@ionolaru4936]

"Starts with an S I'm blanking on it" ? I heard "starts with an S I am plunking on it" LOL, I thought that was just a very nice joke from Tim to suggest Splunk.

[@RudhinMenon]

I couldn’t recollect even when I use splunk 😄

[@haibinxiong1847]

I like this PPT!

[@pajeetsingh]

Do you fork the send of each microservices so that it goes to Kafka as well as to destination.

[@pajeetsingh]

Does one install kafka in every isolated microservice so as to enable it for "streaming"?

[@BrendanMetcalfe]

Great talk thanks!

[@andyvandenberghe6364]

the font is giving me headaches.

[@shellac23]

So how many brokers do you guys usually stand up at your companies? Are you doing this in Azure?

[@dokwme1211]

Just to make it clear 1 Topic 4 Partitions Replication Factor of 3 and yes of-course 4 Brokers

[@ccapeng]

Is leader just like master? Then, follower as slave?

[@BharCode09]

@@ccapeng Yes. Leader is the master. All the Producers and Consumers talk only to the Leader(Master). Followers keep themselves updated by the Leader.

[@maratsahabudinov]

@@BharCode09 Hi. This doesn't seem to be true anymore: stackoverflow.com/a/37829299/997825 cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica

[@RoboRope]

Tell me slow...what is a consumer group? Are they applications?

[@curiosull]

Basically 1 consumer that scaled vertically, and kafka knows not to respond with the same data for a group.

[@sachintiwari1579]

Q1: can a broker manage multiple partitions of same topic? In other words, can no. of broker be less than max no. of paritions in a topic? Q2: If answer to above is 'no' then can we say that in order to add new partition we need to add new broker first? Q3: What is best way to reduce no. of partitions? Should one just delete that extra partition, also will the broker handles deletion of its replica as well? OR is there a way to deactivate a partition I.e. set that extra partition to read only mode?

[@mrNanoDhruv]

Yes, the number of partitions in a topic is independent of the number of brokers.

[@crimsongaze3303]

You can even have a single broker which handles all partition from all topics.

[@balfiman]

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.

[@nigelrmtaylor]

SIEM - I think he was the word he was looking for. But I might be wrong...

[@KrishnaChaitanya]

To add, "Commodity Hardware" here apparently means machines with a minimum recommended RAM size of 64GB. Would I call that commodity hardware? Probably no. But just wanted to point that out.

[@viorelanghel5532]

when you have 8GB RAM on your phone and "normal" servers with 256GB RAM, yeah, 64GB is "commodity hardware"

[@rahulguharay401]

were you talking about Splunk :) , the log processor :)

[@santiagomoneta]

anyone from confluent can add... what changed from this talk to 2020?.. if any..

[@a1988ditya]

TBH ... They should have more experienced person giving these talks ... This guy does not seem to have any hands on real experience

[@kevinm8865]

Typical for a Director of software development in my experience. What is really irritating in this industry is the desire people seem to have to overcomplicate their designs. To use A yiu gotta download and configure B and B requires an annual subscription to X and Y and X depends on T which requires an Azure subscription to store this and that. Is that really the best way to design systems? Give your credit card to a cloud service and pray everything works out?

[@samiullahk7223]

​@@kevinm8865 not long industry is calling out thhe bluff

[@joshreji7510]

7:40 you were actually thinking splunk. But ok.

[@jinandtalsania9303]

Does cloudera official support confluent product?

[@OverG88]

What's the name of tool he used to create those sketches?

[@sgil9985]

paper ios app

[@AnyFactor]

Here I am with a 15 gb json file looking for a versatile db system.

[@briangrange]

Splunk

[@ashishtyagi3638]

Or elastic search :-)

[@BrianDominy]

It was even on one of his slides at 10:19

[@claushellsing]

17:07 What if I use Kafka with just a single partition and a single broker ?. I kind of like of Kafka not just for its distributed nature but for how process streams and Ksql, but also the environment.

[@coryforward7006]

That's fine for local testing and non-critical applications (don't take that to production though), you'd just sacrifice high availability, i.e. if your single broker goes down then your system will be down, you also won't be able to do updates without downtime. Also, with a single node you can still have multiple partitions for a single topic, you just won't have any replication so if your broker has data corruption you'd lose all the data in that partition(s) for a topic.

[@vicky271075]

real Stuff begins after 30minutes

[@bryanbrianbrian]

Unfortunately I did not learn anything

[@khodges3272]

s/form/from/ ??

[@aksheshdoshi]

Also, s/learned/learnt/ ?

[@MrC0MPUT3R]

@@aksheshdoshi Learned is accurate in American English

[@aksheshdoshi]

@MrC0MPUT3R Thanks 🙏

[@srikanthjnr]

This guy is full of himself. Hot 🔥 air. The style of presenting is similar to what a 3rd grade teacher does in primary school.

[@skipperkongen]

Splunk!

[@quirkasaurussaurus2896]

such nice arms on this guy. if only he waved his biceps around more. or wore a tank-top. hair needs a bunch of work though.

[@playdude999]

so in summary: 1. Don't hire administrators that they don't know how to operate Kafka and didn't remember that one of the brokers was upgraded to newer version than the rest of the cluster 2. Poorly designed health check, again - poor administrator or whoever build that and not thoroughly test it. (moral of the story: Hire someone who know how to work with 'sharp knives') 3. And again.. poor guys who runs this cluster didn't paying attention what they are doing.

[@asganaway]

It's always a dev ops fault

[@ifteebuet]

This one is a better insight on particular issues in Kafka production.. kzbin.info/www/bejne/g5nciGR8ismBbqs

[@phanikumaryadavilli2384]

This is one of the worst presentations i came across.

[@sergeibatiuk3468]

Why?

[@phanikumaryadavilli2384]

It is more like unprepared communication. Lots of unwanted taking.

[@phanikumaryadavilli2384]

Infact I was looking for lessons learned not a Kafka lecture.

[@johnsalvatore471]

"splunk"

[@BuvanAlmighty]

Waste of time