awesome video dude!

RabbitMQ is push based model , consumer does not poll , rabbitMq tries to push as soon as it gets the message

what is the software used here to create presentations and interact with Apple Pencil?

Love your channel mate. Very helpful for interview prep

First view 😀

they got cooked on the Tyson fight

Can I ask what should a NoSQL database table schema look like? Should it looks like 0:05 ?

You say consumers when you should be saying consumer groups when talking about Kafka fanout. Otherwise a nice explanation

Great content. It is not clear how to fetch the user's tweet when the key is slowflake id and it is distributed over multiple nodes ?

What about emqtt

Why not Use CDC?

really nice :), thanks

Very good video! Thanks for sharing! One tiny thing, I would prefer NFS over Blob Store like S3 to keep the downloaded pages. A webpage will keep references to lots of resources, like json/css/javascript files. The bold/highlighted words are more important than plain text. That's very important information for ranking. If we don't keep the css files, those information will be lost. So we need to keep them together with the HTML file. It will be very complex to keep those information for multiple files in a single webpage in metadata if we use S3. So I suggest just download the web page with everything to a folder in NFS, and ask indexer to help themselves.

nitpick. We lost quantity when we went from model to table on order_product table

I'm still quite confused on why we prefer queue + Flink (stream processing) instead of just ingesting data directly from sources using spark.

I think you missed a huge aspect about when you to choose between SQL vs NoSQL. At the end of the day, its more about picking the right tool for the job, and less about the scale of the data. If you need ACID transactions, use SQL. If you're okay with eventual consistency then NoSQL might be easier. The scale of your data is just one minor aspect you need to consider when building a system. I'd argue you can scale both types of DBs just as much as each other but it'll all come down to what kind of trade offs you're willing to make. Its not uncommon to have a combination of both NoSQL and SQL DBs, they are not mutually exclusive.

I have faced issues while accessing data in nosql databases. 🥲

Postgresql offers the same power as MongoDB when it comes to json, so there's no sense to choose MongoDB even if you want to store your data in denormalized form.

hi @interviewpen , it's diwali can i get one coupon code so that i can buy your sytem design course, it's diwali , love from india

I have a question. Everytime when a user logs in, the system will generate a session token regardless of if it is a good user or a bad user. How this session token differentiates between the good and bad user?

Good stuff! However, what is also important apart from schema, that relational database has relatively slow writes/updates because of their index structure (B-tree) and storage mechanisms. Apache Cassandra, afaik, has LSM tree and mem tables. MongoDB has async I/O journaling and also has LSM tree (or does it?) in addition to B-tree. I, personally, dont like any relational DB, but "right tool for the right job" :)

this is gold

do most large scale companies use nosql then in favor of ease of storing and accessing data?

I'm i missing something or he didn't design the model in nosql ?

Hey nice video, but I'm confused about 1 thing: You mentioned that the CDNs are caches themselves, but then we're adding an additional cache layer? This seems redundant, no? Even if you go with the requirement of leaving NY data close to NY users, why can't you can still implement this algorithm at the CDN level?

Amazing video, thanks a lot ❤

Great video! Would like to have seen NATs included in the comparison.

What you said is not correct for MinIO. MinIO features a data sharding concept that splits files into specified parts and stores each part on a separate drive.

thank you bro

2:40 is kind of incorrect. RabbitMQ uses at-least-once delivery as it requires messages to be acknowledged by the application (either automatically or manually after doing some work). This way a failure of an app instance which processes a message, but doesn't ack it, or just acks it too late, may result into double processing. The consumer still has to have some level of idempotency to have resulting data in a consistent state.

Have you ever heard about Hermes which is using by Allegro - one of the biggest e-commerce in Poland

So the BLOB system assume every node have the exact same storage capacity ? Does it do rehashing if a node is already full and cant accept more data ? What happen if a node or multiple nodes are down ? What happen if a node have too many connections for reads ? Do the network congest or does it load balance in a specific way ? Are the reads cached somewhere if some datas are way more accessed than others ? If multiple clients fetch the same data at approximatively the same time (in the same window of time), are both queries optimized to use less ressources together ?

which whitebaording software do you use?

Very nice vid! Great job explaining something this complex in 5 mins! A minor note is that S3 clients don't know about the topology behind the scenes, so it is not the client, but "something" in between. For MinIO it is the server itself that does the hash and reaches out to the nodes with the data and collects it to serve the request. But the principle is exactly as you described! (I work @ MinIO) Having the requester know the remote host topology makes the clients more complex, and also makes serving via reverse proxy/load balancing hard. But of course you avoid a forwarding hop, so pros and cons. Memcached, Aerospike does this for example.

It would be cool to have a video on design of url shortener

Thank you for a nice details explanation @ mechanisms of Kafka

What software is he using to draw?

please keep doing these system design videos where you go over a description of operations and explain the architectures behind them. this is great

could you explain please how different Kafka consumers are assigned to this or that consumer group? I guess this is the key moment in creating the balanced model with multiple consumers in Kafka

what exactly do you mean by "consumers in RabbitMQ have control over what messages they are consuming"? either they are binded to direct queue or topic they always consume messages - aren't they?

strange... AMQP (and RabbitMQ) has topics that let distribute messages to multiple listeners as well. It was never mentioned

Honestly one of the best videos on the internet for system design!

Бери кафку - кафка фкуфнее! 😆

Are we basically talking about queue vs topic?... Those are just implementation of each type no?

I think it would be great to have a follow up video covering AWS IoT core

1:44 Haven't you heard pub sub in rabbitMq? RabbitMq delivers only one message to each consumer is such a wrong statement dude ... This exact use case i had implemented 8 years ago with rabbit

Hi, what exactly is this subject? Is it data science?

I thought the whole idea of Google Drive and Dropbox was to focus on distributed storage rather than just using ready-made cloud solutions like S3. In my opinion, that's the most interesting aspect of such a task.

Haven't all moved to pulsar yet?

Some parts were misleading and confusing unlike the other videos. I'd suggest to redo this video to match the quality with the other videos :) , thanks for all the great contents