3. Apache Kafka Fundamentals | Apache Kafka Fundamentals

  Рет қаралды 432,157

Confluent

Confluent

3 жыл бұрын

cnfl.io/apache-kafka-101-lear... | In this video we’ll lay the foundation for Apache Kafka®, starting with its architecture; ZooKeeper’s role; topics, partitions, and segments; the commit log and streams; brokers and broker replication; producers basics; and consumers, consumer groups, and offsets.
After you’ve watched the video, you can take a quick quiz to check what you’ve learned and get immediate feedback here: forms.gle/RDc84FbPeJ2CwCRP9
As always you can visit us here: cnfl.io/kafka-training-certif...
LEARN MORE
► Apache Kafka 101 course: cnfl.io/apache-kafka-fundamen...
► Learn about Apache Kafka on Confluent Developer: cnfl.io/confluent-developer-t...
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka, enables organizations to harness business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit confluent.io
#apachekafka #kafka #confluent

Пікірлер: 125
@Adi-mj3cb
@Adi-mj3cb 3 жыл бұрын
This is a gem of a video series with an even greater gem of a presenter. I truly mean this when I say that Tim is just insanely good at breaking down info like this. He deserves all the promotions and love that can be humanely offered by any corporation on this planet. I encourage him to keep making videos like this on more topics. My sole regret is that this video did not come out years earlier. Tim, if you're reading this, I applaud you. You are unironically my hero of this month and I hope you continue living a great life.
@franciscos.2301
@franciscos.2301 Жыл бұрын
Yeah, Tim's very charismatic, approachable and well-spoken. Most people aren't. Respect.
@desi4masti
@desi4masti Жыл бұрын
I felt same way and came here to write same . Saw your message so just "Reused". Loud claps for Tim.
@DonJuanDM
@DonJuanDM Жыл бұрын
I fell asleep on most other technology videos but Tim is exceptional good. His skill on verbal fluency and the use of context keeps me wide awake.
@khrisna6435
@khrisna6435 2 ай бұрын
absolutely agreee!
@SunsetNova
@SunsetNova 12 күн бұрын
Tim you’re the best
@mikesunny1291
@mikesunny1291 3 жыл бұрын
official and legit man! I just wondering why this kind of official tech videos are not listed at the top when searching "what is kafka". bunch of superficial and low quality videos show up and contains wrong information and non-core details. really hope you guys PR these videos to let more engineers get the correct stuffs.
@deanwest5581
@deanwest5581 Жыл бұрын
This guy is the best tech educator I've ever seen.
@somnathhazra6959
@somnathhazra6959 2 жыл бұрын
Nobody ever taught Kafka in details like you did Tim. Much appreciated 👍🏻
@alexeyelivanov1704
@alexeyelivanov1704 3 жыл бұрын
Thank you, Tim and the Confluent team, your tutorials are top-notch and really help people understand the subject. Wish you all the best!
@SilviaTosi
@SilviaTosi 2 жыл бұрын
Entering now in Kafka and finally a perfect simple rapid nice explanation. Thank you for this video!
@sp-sj4st
@sp-sj4st 3 жыл бұрын
Thanks Tim for the awesome explanation of Kafka terms and how they relate with each other. After watching this video only I could understand kafka terms in real deep and I no more have to cram these terms again amd again.🙂
@RobinSingh-md1sh
@RobinSingh-md1sh 2 жыл бұрын
Halfway through the video and already loving this guy. Super presentation and delivery skills. Kudos Tim!!
@kant.shashi
@kant.shashi 3 жыл бұрын
this is exactly i was looking for.. best of all the videos on Kafka Fundamentals
@sheetaljigi
@sheetaljigi 2 жыл бұрын
Excellent explanation in a short video Tim. Really appreciate your effort. Thank you!!
@kitnguyen7259
@kitnguyen7259 2 жыл бұрын
First time learning about Kafka, learned everything I needed to know from your one video. You are a great presenter sir, thank you!
@Th3Arock
@Th3Arock Жыл бұрын
That was one of the best description of Apache Kafka fundamentals that I saw, thanks
@hugpic7812
@hugpic7812 2 жыл бұрын
Super course, very effective and well delivered. Thanks !
@NeerkumarOOPS
@NeerkumarOOPS 3 ай бұрын
Straight forwared, right on point, no bullshit. Cleared all main kafka concepts in just 24 mins.
@lucak.2138
@lucak.2138 Жыл бұрын
Just wanted to express my appreciation for this (university level) lecture! Thank you, very well done!
@chris0628
@chris0628 11 ай бұрын
Excellent content 👍🏽 to the point, no fluff, clearly explained with diagrams 👍🏽
@hello-again6994
@hello-again6994 2 жыл бұрын
I can't believe I just watch a 24 minute technical video without yawning or pausing. This guy is good! 🏆🏆
@firingpistonz
@firingpistonz 2 ай бұрын
The Best Kafka fundamentals video. Period. Give the instructor an award. So precise and to the point.
@kal6392
@kal6392 8 ай бұрын
Excellent content 👍🏽 to the point, no fluff, clearly explained with diagrams 👍🏽 Thank you.
@jenithmehta9603
@jenithmehta9603 7 ай бұрын
Hi Tim, thank you for a great explanation. Also thumbs up to the team behind the video.
@skywalker66ful
@skywalker66ful 2 жыл бұрын
One of the best videos on Kafka basics and understanding of the Cluster
@sambo7734
@sambo7734 2 жыл бұрын
Hey :) Your presentation and explanatory style is really great - excellent in fact, and just at the right level for me! Many thanks
@geethaachar8495
@geethaachar8495 8 ай бұрын
i can state unequivocally that i have never come across a better video instructor.. Hats off
@amitjaini
@amitjaini 2 жыл бұрын
Loved it. Thanks for making it easy to understand.
@enver6414
@enver6414 Жыл бұрын
I can say that I am not watching any other channel for Kafka now.good job!!!
@sunnyp219
@sunnyp219 Жыл бұрын
Amazing video. Very thorough. Can't believe this is free. Thanks Confluent.
@bhanupkatta
@bhanupkatta 2 жыл бұрын
This guy’s presentation skills should be made gold standard, period !!!
@hetalchavan7379
@hetalchavan7379 2 жыл бұрын
Great video! Simple to understand and managed to hold attention
@abdulahadkhan705
@abdulahadkhan705 Жыл бұрын
Much appreciated! Thanks. Highly recommended for freshers with couple of years of experience, starting journey towards resiliency and kafka.
@adityagoel723
@adityagoel723 26 күн бұрын
Such clarity of thoughts and impeccable explanation, answered all the questions that came to my mind. Best Kafka video!
@pwb83
@pwb83 Жыл бұрын
Excelent tutorial! Straight to the point and super clear
@chillbro2275
@chillbro2275 Жыл бұрын
Man your rocked this presentation! Thank you Tim.
@NirajKumar-Geeks
@NirajKumar-Geeks 3 жыл бұрын
Very well explained. Thank You so much :)
@kenjimiwa3739
@kenjimiwa3739 5 ай бұрын
This speaker is great, with a casual style and different intonation that's easy to follow. In a lot of tech KZbin videos the speaker just rambles on in a monotone voice making me want to zone out.
@nguyenshmily7118
@nguyenshmily7118 2 жыл бұрын
Awesome, Very simple to understand and managed to hold the attention
@GauravKumar-nh2ii
@GauravKumar-nh2ii 10 ай бұрын
Such great content and so much detail oriented that it's cleared my all doubts. Thanks for making such content. 👍
@AndrewPerry1
@AndrewPerry1 3 ай бұрын
Great that the day has come that Zookeeper is no longer required - nice that you were able to give a heads up!
@olemaiwald1873
@olemaiwald1873 3 жыл бұрын
great video and even better series!
@sajidmohammed8665
@sajidmohammed8665 Жыл бұрын
Tims explanation!...top notch !
@cltan8
@cltan8 Жыл бұрын
Thank you for the video. Helps me a lot to understand the components of Kakfa.
@radhianand5647
@radhianand5647 Жыл бұрын
Amazing.. crystal clear explanation Tim.. Thanks a lot ..
@ApoorvaGarg0308
@ApoorvaGarg0308 3 жыл бұрын
Very well compiled, i struggled a lot and you have put all information at one place! cheers!
@danielhaile9073
@danielhaile9073 3 жыл бұрын
Thanks great breakdown and presentation .
@michaelkazerooni1019
@michaelkazerooni1019 Жыл бұрын
This was awesome! top-notch! Bravo
@anjildhamala4996
@anjildhamala4996 2 жыл бұрын
This was solid. Thank you!
@nk91able
@nk91able Жыл бұрын
its a great video to understand the basics, thanks for posting this
@sanzharsuleimenov6380
@sanzharsuleimenov6380 Жыл бұрын
Explanation is gorgeous !
@itsmendyou
@itsmendyou 5 ай бұрын
Solid knowledge, thank you fro sharing this.
@rajeshpadmanabhan8443
@rajeshpadmanabhan8443 2 жыл бұрын
Thanks Tim for this excellent video.
@madhousetoobah
@madhousetoobah Жыл бұрын
Top Notch explanation/video!! Well done!
@naeemsidd6533
@naeemsidd6533 9 ай бұрын
Awesome presentation, crystal clear
@mwont
@mwont Жыл бұрын
Awesome explanation. Thank you
@andybhat5988
@andybhat5988 2 жыл бұрын
Excellent explanation. Thank you
@dalgonacovfefe
@dalgonacovfefe Жыл бұрын
this is such a great explanation thank you
@user-gp5xz
@user-gp5xz 11 ай бұрын
Great video, thanks alot!
@enjoyhiking9845
@enjoyhiking9845 Жыл бұрын
Very good presenter and very informative video, thanks for sharing! 😄
@sarveshlohani
@sarveshlohani 3 жыл бұрын
Nicely explained. Thanks
@irfanbabar8424
@irfanbabar8424 5 ай бұрын
Great explanation with good detail. thankyou for your great effort.
@ytdlgandalf
@ytdlgandalf Жыл бұрын
Nice intro! Other complicated software should take note of this simple introductionairy video
@zaidwaqizulkifli6554
@zaidwaqizulkifli6554 2 жыл бұрын
Dear Confluent, Can we have Tim do videos on Spark, Druid and Kubernetes too?
@varungupta6596
@varungupta6596 3 жыл бұрын
Well Explained. Question : How the Disk Space underneath the Brokers/Segments Grow? Is that something Producers or Consumers need to be worried about? That's Cloud Offering as a SaaS or IaaS?
@zambishon
@zambishon Жыл бұрын
fantastic video! BTW I like the shirt 😎
@MaheshPatel-om5vq
@MaheshPatel-om5vq Жыл бұрын
Tim, You are very good explainer bro
@hannnah689
@hannnah689 2 жыл бұрын
Great talk!!
@robotempire
@robotempire 2 жыл бұрын
AMazing videos. Consider using patterns or other visual indicators in place of color to accommodate various color-blindnesses
@thamsanqamngxunya4460
@thamsanqamngxunya4460 10 ай бұрын
well explained thank you Sir
@kalkalasch
@kalkalasch Жыл бұрын
Wow this is easily understandable and not boring. Is it too much to ask to make videos of this quality of other topics in tech..?
@Anonymous-hp1tg
@Anonymous-hp1tg 2 жыл бұрын
Cool, Unbelievable such a valuable information is for free.
@JohnnyMagorish
@JohnnyMagorish 3 ай бұрын
Very well explained. My only gripe is that I see that more as an azure blue
@jacobborusu1923
@jacobborusu1923 3 жыл бұрын
Can the offset be synced across partitions so that we can have serial processing of the data ?
@EwertonSilveiraAuckland
@EwertonSilveiraAuckland Жыл бұрын
awesome video. Thanks
@pratheek5096
@pratheek5096 2 жыл бұрын
Who is this guy ? Blew my mind away
@mostinho7
@mostinho7 Жыл бұрын
Done thanks Logs are immutable, can have multiple consumers consuming from different offsets of the log because consuming doesn’t actually delete the messages from the log Partitions of a topic live on different brokers and not all brokers must have partitions from a topic Partitions are replicated across brokers, so that if broker fails the topic partition is replicated elsewhere and can change the replication factors for partitions. Master/leader for each partition when it’s replicated and writes happen at the leader CLI producer can be used for testing How does a producer know which partition to write the message to? Partitioning strategy by default uses the hash(key) % numpartitions Messages with the same key will land in the same partition and will maintain their ordering. (In the case that number of partitions in a topic changes then this isn’t the case anymore but they shouldn’t change) Each consumer has an offset of where in the log it’s reading from. Consumer pulls messages from topics after offset n
@vkp2001
@vkp2001 Жыл бұрын
love the video
@georgetsiklauri
@georgetsiklauri Ай бұрын
8:50 One slip here. I think it should be "the broker, that topic lives on" and not "that partition lives on", because 1) you haven't even introduced partitions yet, at this very moment, and 2) you later say "if you're writing messages into that topic".. which again confirms my point, that you mean topic, not a partition.
@juvenilemrcia2664
@juvenilemrcia2664 2 жыл бұрын
When he mentioned timestamps, I was imagining a cop saying “You have the right to a time stamp. If you cannot afford a time stamp, one will be provided for you.”
@masterprattu
@masterprattu 2 жыл бұрын
Fantastic video @Confluent and @TimBerglund. Confluent is amazing in the way that they are making kafka easier for everyone to learn.
@agustindearmas473
@agustindearmas473 2 жыл бұрын
Thanks man
@chetan2399
@chetan2399 2 жыл бұрын
Does Kafka topics got in built filters to filter out messages ?
@leovoldoftrest9987
@leovoldoftrest9987 2 жыл бұрын
Very good
@umatjeet8012
@umatjeet8012 2 жыл бұрын
wowwwwwww this video is sooo freakinggg goood
@dnbndu
@dnbndu 2 жыл бұрын
Hi, I am fresher, and new to Kafka. For storing those topics we need persistent storage and consuming a message don't delete that message. Now my question is suppose I have X ammount of persistent storage and producer produces X ammount message in Y days, what will happen to my storage after Y'th day?
@vincentbuscarello1357
@vincentbuscarello1357 3 жыл бұрын
Very helpful. As a bizarre side note, the speakers voice sounds a lot like Weird Al Yankovich to me. Which is obviously a very good thing.
@garhhh9513
@garhhh9513 Жыл бұрын
10:55 Cornflower blue, you say, CORNFLOWER BLUE ??? I'm shocked. SHOCKED to find out that colors aren't being identified in here" Why, that is clearly a case of #7cb0f9, which as anyone know is soft blue! Thank you for the wonderful video. You, sir rock. _List below is for me to quickly find stuff. Be warned, these time points don't do justice. Watch the whole video. It's awesome_ 0:02 - Nice little joke here 0:09 - Nice upbeat music. Don't miss this one! 0:20 - Synopsis of what you will learn in 24 minutes 0:54 - Kafka's job 0:59 - What's a producer 1:33 - Data stuff that goes into a Kafka Cluster 1:57 - What's in a Kafka Cluster ? 2:21 - Brokers 3:32 - Consumers 3:57 - Relationship Consumer and Application 4:15 - Reiteration of Fundamental parts of Kafka 4:52 - ZooKeeper 5:45 - Decoupling of Producers and Consumers 6:51 - What does ZooKeeper really do in a Kafka Cluster ? 7:39 - Topics 9:11 - Partitions 10:19 - Segments 11:01 - Topology and Detail example of a Kafka Cluster 12:25 - What's a Log 14:02 - Consumers. Do they really consume ? 14:55 - Structure of a Kafka Message 16:10 - Brokers revisited 17:19 - Broker Replication 18:16 - Producers revisited 19:51 - Load Balancing 21:25 - Consumers revisited 22:41 - Distributed Consumption
@Draaack
@Draaack Жыл бұрын
This guy PRESENTS.
@DNI404
@DNI404 11 ай бұрын
thanks for the video, but you made a mistake about the colors: #71cc01 - not green #ee9f00 - not orange #7bb1fe - not cornflower blue
@justmeandmy
@justmeandmy 2 жыл бұрын
Wait, but how do the consumers of the consumer_offset topic keep track of where they are?
@gagyboki799
@gagyboki799 2 жыл бұрын
What s the maximum file size that Kafka can process?
@njg120
@njg120 Жыл бұрын
why kafka use consumer and suscriber in their terminology ? this are different approach in messaging , so what Kafka model use ? consumer/producer or publish/suscriber ?
@avr4dev
@avr4dev Жыл бұрын
At 4:53 why you have `n` everywhere? That might be confusing since it is a kind of implication that we have equal number of components on all the levels.
@djkim24601
@djkim24601 Жыл бұрын
I came for information and received quite some laughs as well.
@alexanderwitte9919
@alexanderwitte9919 2 жыл бұрын
hahaha cornflower blue. awesome
@sharmarahul17
@sharmarahul17 11 ай бұрын
👌
@themodernarchitect7537
@themodernarchitect7537 2 жыл бұрын
21:24 now you confused me. I thought that the consumers used long polling, but you described a short polling mechanism.
@Jason-ky4ue
@Jason-ky4ue Жыл бұрын
@10:56 It is cornflower blue. I know this because Tyler knows this.
@no_more_free_nicks
@no_more_free_nicks 2 жыл бұрын
You can say that a broker is a process?
@yushutong722
@yushutong722 2 жыл бұрын
Thanks for this great video! I have a few questions that hopefully someone could help clarify :) 1. How are brokers replicated? It sounds like it's async replicated, hence I imagine when the leader failover there would be some small amount of msg loss (because replica would always lag primary a tiny bit)? 2. When the broker receives the msg, does it write to log immediately or does it do some kind of in-memory buffering and write by small batch? And if so what happens to the non-flushed messages if that broker crashes? Just to clarify, I'm not criticizing Kafka, it is a great tool and I really liked it while working with it in my previous job. But I'm just curious because I've heard various techtalks about how kafka is used in various products -- e.g. nu bank which is a financial startup uses kafka and from their talk I had this feeling that they rely on kafka for being 100% reliable (as in, not losing messages), which surprises me. 3. One last noob question, apparently Kafka's great capability for supporting high write throughput is partially due to its sequential write, hence avoiding random disk seek. But given that consumer almost always consumes the message with a slight delay, does it mean whenever a consumer pulls new messages it breaks this nice sequential mechanism (because we need to seek to a different disk location than the end of the log file)? Thanks!
@yushutong722
@yushutong722 2 жыл бұрын
To answer question #1 (after watching the next video in the series), apparently one can tune the replication iin Kafka to only have producer acked after the msg is replicated to all replicas.
@tejeshreddy6252
@tejeshreddy6252 Жыл бұрын
Hey have you found the answers to your questions #2 & #3? I don't think any system can be 100% reliable but considering industry wide adoption I believe they must be doing something really well. I would love to know their solution to these fundamental data problems
@ingoos
@ingoos Жыл бұрын
i suppose that Kafka is like those newsstands where some of us still go to to get a newspaper / magazine
@TheEsisia
@TheEsisia Жыл бұрын
I had no idea Bill Burr is so good with Apache Kafka stuff as well...
@huskyrbx1318
@huskyrbx1318 8 ай бұрын
Kafka is a mommy wait I think I went to the wrong vid 😑
@stealthragnarr2644
@stealthragnarr2644 2 ай бұрын
Kafka is Mommy of ETL
@SubhamDas-tb3xf
@SubhamDas-tb3xf 2 ай бұрын
Yes
4. How Kafka Works | Apache Kafka Fundamentals
26:41
Confluent
Рет қаралды 181 М.
What is Apache Kafka®?
11:42
Confluent
Рет қаралды 329 М.
How to open a can? 🤪 lifehack
00:25
Mr.Clabik - Friends
Рет қаралды 8 МЛН
FOOTBALL WITH PLAY BUTTONS ▶️ #roadto100million
00:24
Celine Dept
Рет қаралды 147 МЛН
ФОКУС С ЧИПСАМИ (секрет)
00:44
Masomka
Рет қаралды 3,9 МЛН
Суд над Бишимбаевым. 2 мая | ОНЛАЙН
7:14:30
AKIpress news
Рет қаралды 622 М.
Event-Driven Architecture (EDA) vs Request/Response (RR)
12:00
Про Kafka (основы)
49:23
Владимир Богдановский
Рет қаралды 341 М.
Microservices explained - the What, Why and How?
18:30
TechWorld with Nana
Рет қаралды 775 М.
Docker Crash Course for Absolute Beginners [NEW]
1:07:39
TechWorld with Nana
Рет қаралды 1,3 МЛН
Kafka Tutorial - Core Concepts
13:04
Learning Journal
Рет қаралды 905 М.
System Design: Why is Kafka fast?
5:02
ByteByteGo
Рет қаралды 1 МЛН
host ALL your AI locally
24:20
NetworkChuck
Рет қаралды 112 М.
What is Kafka?
9:17
IBM Technology
Рет қаралды 432 М.
Распаковка айфона под водой!💦(🎥: @saken_kagarov on IG)
0:20
Взрывная История
Рет қаралды 10 МЛН
План хакера 🤯 #shorts #фильмы
0:59
BruuHub
Рет қаралды 996 М.
Компьютерная мышь за 50 рублей
0:28
dizzi
Рет қаралды 1,7 МЛН
Samsung or iPhone
0:19
rishton_vines😇
Рет қаралды 269 М.