Design a High-Throughput Logging System | System Design

[@wizz0056]

Kafka -> Loki -> S3 If you're looking for an existing solution :)

[@interviewpen]

Yep, S3 does a lot of the things discussed here behind the scenes. Thanks for watching!

[@lunaxiao9997]

great video，very clear

[@interviewpen]

Thanks!

[@developerjas]

Great Video man! Would how would you go about designing the data ingestion part?

[@interviewpen]

Great point! There’s a lot that goes into ingesting logs while optimizing network performance and maintaining context. Check out our full video on monitoring systems on interviewpen.com :)

[@sahanahunashikatti3935]

😊😊 ok 0​@@interviewpen

[@supragya8055]

i dont understand , if under same bucket lets say for (2021-2022) we have multiple nodes , how are reads any faster ? for the same bucket logs will be distrubuted across servers and still need to be queried across servers which is slow . Bucketing didnt help in improving read performace , is my understanding .

[@interviewpen]

Yes, sharding improves write performance at the expense of query latency (unless we shard by something more clever!). However, we can still handle a high throughput of reads. This latency vs throughput problem is a common tradeoff with large-scale systems! Hope that helps :)

[@sahanagn4485]

Great video!!! Please slow down the speed of video as someone new to topic its bit fast to grasp the concept.

[@interviewpen]

Ok, noted!

[@GoofGoof-cs6ny]

So in 2018 every service was writing logs to node 3, didn't we went back to bad write complexity by doing bucketing?

[@interviewpen]

Yep, bucketing makes query performance better, so we introduce sharding as well to distribute writes within a bucket.

[@michatobera6049]

Great video

[@interviewpen]

Thanks!

[@didimuschandra6680]

Greatt video!! thanks! but, can you create video to develop Effective and efficient Ticketing System?

[@interviewpen]

Sure, we'll add it to the backlog. Thanks for watching!

[@prakharsrivastava6644]

I love the cute computer in the background

[@interviewpen]

Thank you :)

[@weidada]

Suppose every two years, it ingest 2PB and migrate 1PB, how could three sets be enough to cycle after 12 years?

[@interviewpen]

Great question! At any given time, we have three "hot" nodes--two are migrating data to cold storage and one is ingesting new data. We only showed one cold storage node in the example, but we would need at least 2 to make this work long-term. Hope that helps!

[@ankushraj3599]

Why not use Kafka for high through put?

[@interviewpen]

Kafka is an event streaming platform, so it wouldn't solve any of the log storage problems we're addressing here. But if you have any thoughts on how to incorporate it, feel free to share!

[@RaushanKumar-co3wj]

@@interviewpen Use kafka stream + cassandra . process the event through consumers and save inside a Hbase db for analytics .