Thanks for the videos. I think one important functional requirement that most logging solutions offer ( GCP logging ...) is text search. So potentially having a text search engine (ELS) is something to consider.
@jordanhasnolife5163 Жыл бұрын
Makes sense! I think that's a reasonable thing
@rahulaga2 жыл бұрын
best part is that you really focus on 'why' aspects of things, keep rocking !!
@jordanhasnolife51632 жыл бұрын
I appreciate it man! That's why I originally made this channel!
@Baronvonbadguy32 жыл бұрын
Same reason i love these 👌
@art4eigen932 жыл бұрын
That intro. .. 🤣🤣🤣
@VyasaVaniGranthАй бұрын
My suggestion to keep this channel going would be to get ripped and document your fitness journey. Just a thought :D
@jordanhasnolife5163Ай бұрын
Haha it's been almost 2 years since I uploaded this video, some stuff has changed
@VyasaVaniGranthАй бұрын
@@jordanhasnolife5163 Suggestion still holds ;)
@saisreenath41993 ай бұрын
High quality content. Keep doing these videos 👍
@advaitchabukswar41636 ай бұрын
This video really helped me in one of my interviews, thanks a lot!!
@jordanhasnolife51636 ай бұрын
Hell yeah!
@baby_medusa77722 күн бұрын
Sense of Humor A1
@theghostwhowalk2 жыл бұрын
One more great video!! Loving it!
@jordanhasnolife51632 жыл бұрын
Thanks!
@prashantbharadwaj63412 жыл бұрын
Good one Jordan. You are very clear in your thoughts. Keep this going !! :) Metrics/Logging system is challenging because of both high scale writes/reads. It looks like the write scale here depends on how much Kafka can scale. If we are looking at a very active public service receiving 100 billion msgs/day (1000 msgs/sec), I am guessing Kafka can handle that ? What about read load ? Since lot of people may use the log for customer investigations, there could be a lot of read load on the time series DB since the other path is for batch insights. As I am typing this, I am thinking about splunk. Could you make a video on how to design splunk like system ? (May be these are the building blocks)
@jordanhasnolife51632 жыл бұрын
Spunk is just a distributed search index as far as I know, which I already have a video on. It's called twitter search/elastic search, which you could just connect to kafka.
@raysdev Жыл бұрын
awesome content! sorry for skewing your metrics towards the other 97% but that was funny as ...! 😅
@jordanhasnolife5163 Жыл бұрын
Haha no worries I'm grateful for all my viewers regardless of gender
@bryanbrianbrian13 күн бұрын
I quite hate system design interviews and regurgitating proper nouns I’ve never engaged with, think I’ve chosen the wrong career
@jordanhasnolife516313 күн бұрын
Option 2 is to engage with them
@helperclass87104 ай бұрын
Thanks for the amazing video. In one of the interview I was asked to design flight recorder to record the data within a flight. Could you please make a video on that.
@jordanhasnolife51634 ай бұрын
Interesting - so fwiw flight recordings are just taking various samples of the program at slightly randomized intervals, so I think that the design would be very similar to this one, but you can just partition events based on the thread they were published from in your time series database
@tavneet199412 ай бұрын
Hey, how about using Apache Pinot or druid to support better querying capabilities directly on the real time data?
@jordanhasnolife51632 ай бұрын
Hey, I'd have to look into these more, but if it allows you to query directly from kafka itself in addition to the db then I suppose that could speed things up a bit. Depends whether we need it!
@dind79269 ай бұрын
hey Jordan, what prevents us from sending the unstructured data directly from the client to the S3? If we do not care about data enrichment we might as well just send it straight from the client, unless I'm missing something? also a couple of follow up questions just to clarify it for myself: - why do we need a logging service, why can't we just push the data from the client straight to the queue? - as far as I understand we leverage Timeseries DB for queries on relevant "recent" data, so I assume we would need some sort of clean up jobs that run periodically? And we use data warehouse (like Snowflake) to enable analytical queries that would be too big to run on our main DB?
@jordanhasnolife51639 ай бұрын
1) I agree, though in this case I was assuming that we are doing some sort of data enrichment via the flink consumers. 2) Similar to your first question - but at the end of the day there are reasons we want to have some sort of gateway before we process every request to send data to kafka. Examples might be rate limiting or some sort of validation on the messages to ensure that a bad actor can't spam our kafka queues. 3) Yep! Timeseries DBs making dropping old data very simple, that's one of the main benefits of them.
@ShreyaGupta-nc7td4 ай бұрын
Thank you for the amazing content! Can we use Cassandra instead of S3? What would be the trade offs?
@jordanhasnolife51634 ай бұрын
see previous comment to duplicate question
@calvio2835 Жыл бұрын
Once we have the data in the time series DB, how do you suppose we go about hooking up a monitoring/alerting service to it? I'm not sure what the optimal route is between 1. push based model where for every new metric (or batch) in the time series DB, we query an alarms/rules DB, or 2. pull based model where the alarming service periodically queries the time series DB for all alarms/rules in the DB. 1 seems excessive since majority of real time metrics aren't going to fire an alarm. 2 seems excessive in that most alarms aren't firing at a given instance.
@jordanhasnolife5163 Жыл бұрын
I feel like 1 is probably more practical, especially since you can just build an additional change data capture queue off of the time series db to read from an alarm, but truthfully am not entirely sure.
@ShreyaGupta-nc7td4 ай бұрын
Thank you for the amazing content! Instead of S3, can we use Cassandra? what would be the trade offs?
@jordanhasnolife51634 ай бұрын
I suppose we could - though unless we want structured data not sure why we would (I'm assuming these are just logs for now). Presumably it would be more expensive to run a managed cassandra db than just dumping in s3.
@TheImplemented Жыл бұрын
Thank you, how did you manage to grasp systems design in such a short time? What is your approach of studying?
@jordanhasnolife5163 Жыл бұрын
Hi! I still have much to learn, but my approach was to read as much as I could, starting from Designing Data Intensive Applications. From that point on, whenever I'd see a piece of technology I hadn't heard of I'd make note of it and look it up later. I also take notes on all of this so I retain the information better.
@rajrsa Жыл бұрын
Thanks for this! Is flink consumer just like a normal java/spring queue consumer that is monitoring a AWS kinesis stream? (I've never used flink/kafka.) Do we have to use flink in conjunction with kafka queues or would any service work?
@jordanhasnolife5163 Жыл бұрын
I have a dedicated video about this, but pretty sure flink is flexible with multiple types of message queue. Ideally though you should be using a replayable one though.
@user-db5mx5ou4s Жыл бұрын
can we use HDFS instead of S3? that way we'll achieve data locality and will be part of hadoop cluster? - will be cheaper as well?
@jordanhasnolife5163 Жыл бұрын
From my understanding it's not necessarily cheaper if you're just using it for data storage - if you want to do processing of the data that's different - not sure we need to do any in the Hadoop cluster here though
@Piyush-ky9ee2 жыл бұрын
How can your single leader replication in TimeSeries DB handle the enormous amount of writes ? Won't it be overwhelming for that single leader ?
@jordanhasnolife51632 жыл бұрын
Sorry if I didn't specify, you can shard based on the source of the metric
@Piyush-ky9ee2 жыл бұрын
@@jordanhasnolife5163 But if you do that won't it create Hotspots/Noisy neighbor problem ?
@jordanhasnolife51632 жыл бұрын
@@Piyush-ky9ee It would but since we're just ingesting the data it's really not a big deal in my opinion - if you split the TSDB by the source of the metric and just say send all of the metrics to that one shard of the TSDB I don't think that it should be overwhelmed, as the writes are being buffered by a queue and the entire index can be cached for fast ingestions - perhaps I'm wrong
@sumeet2707 Жыл бұрын
Hi Jordan - Thanks for this video. Do you mind sharing which Pinterest video you referred to in this design?
Kafka and spark are used for different things - Kafka at its core is a message queue, whereas Spark is a big data engine for batch processing. Although there may be kafka consumer software now, when I hear Kafka I think of a message queue.