How Discord Stores Trillions of Messages | Deep Dive

[@hnasr]

Fundamentals of Database Engineering Course database.husseinnasser.com

[@harriehausenman8623]

Just one small note: I think you should really like (♥) some comments, old al-Khwarizmi really seems to like that 😉

[@HadiAriakia]

Sorry mate, already purchased 😂 virtually on the release date. I definitely recommend it to anyone interested to be a good data engineer.

[@TylerTriesTech]

This has to be one of my favorite videos on your channel. Your live reactions and excitement about this stuff is fun to watch.

[@thatguyadarsh]

What a great in depth thought provoking awesome analysis! Thanks for all your work Hussein. Much appreciated!!

[@D3FKONMusik123]

The best youtuber to watch at 1.5x speed

[@fleap]

🤣

[@thedankest1974]

This guy talks way too slow. Like get to the point bro

[@brucewayne2480]

​@@thedankest1974 this is my problem with hussein , even though his content is very interesting but I wait a lot to hear something new and his videos are so long 😛

[@AbhinavKulshreshtha]

At 1.5x, over 80% of the video felt like a normal speed video. You won't miss anything.

[@alexquix6394]

I really appreciate that he talks slow, I am not native speaker

[@kooperl]

Blows my mind that this stuff is just on the internet for free (especially the blog) (especially Hussein's analysis)

[@nishantgoel769]

Superlike.... Your way of narrating the article makes it like a Nolan Movie. love the way you deliver this article. Thanks Hussein for creating such wonderful videos

[@AkashKamal1998]

The discord series of yours taught me alot. Thank you so much.

[@woolfel]

basically discord implemented smart driver to fix the issue. One long standing issue with the original thrift drivers is it wasn't smart. That caused a lot of IO thrashing and especially under load if there are hot partitions. When datastax introduced a newer protocol for non-thrift drivers, it was primarily to fix async read/write issues.

[@woolfel]

a common recommendation for Cassandra is date based UUID. This goes back to pre 1.0 release, so it's preferred over random UUID. Using Cassandra as a database for messaging system is a known anti-pattern going back to version 1.x days.

[@harriehausenman8623]

Amazing content! I like the setting, the voice is nice and calm , you take your time to think inbetween sentences (rare skill!! 😆) and the general awareness of speech is crucial for computer science (and handywork). Thanks so much! 🤗 Oh, and the actual content/deep-dive is on-point.

[@davidtheprogrammer]

This is fantastic! Thanks for the deep dive, learned a lot

[@TheNayanava]

Amazing stuff!! I think memtables are stored in the form of RB trees or AVL trees, so they are already sorted, they are then serialized and stored in the form of sorted string tables. Second point additionally on compaction, it is not very efficient to perform compactions because it steals CPU cycles, and would pre-empt serving actual user requests.

[@iyxan23]

I really love the deep dive format of these videos, I just wanted to give a bit of a suggestion: can you talk a bit faster and like get rid of delays? I occassionally get bored being too impatient about what you're going to be explaining. It's a bit of a shame for me that these very interesting content becomes boring just because of the way you talk. Anyways, keep up the good work 👍

[@0xc0ffee_]

Can I pay you for lessons/coaching? You're the most amazing teacher I've ever listened to. :O

[@willi1978]

I guess they could use Scylla/Cassandra in analogy since Scylla is a rewrite of Cassandra that claims to be 10x faster

[@xslvrxslwt]

because it is. everything that uses Java is so bad 💀

[@sj82516]

Awesome deep dive into the article. I learnt a lot more comparing to read the article alone. I bought your courses and look forward to learn more on DB. I got one question I would like to ask. about the data migration, I think the dual writing is quite tricky. It cannot guarantee the data consistency. I am wondering how the make sure the data is exact the same in Cassandra and ScyllaDB, which means how they guarantee the dual write must fail or success all at once.

[@woolfel]

The old RDBMS master-slave design doesn't scale well for global distributed platform. Even if you use transactor design like Datomic, you can still overwhelm the cluster. Then there's designs used by data grids like Coherence. Scaling a database to trillions of message is tough. Even oracle RAC would have a tough time scaling for this type of load.

[@fernandozago]

This is a great use case for testing any actor model systems out there for "caching" and "coalesing" calls... =)

[@fs811523]

Although I had the same question about why not use cache? I think a message can be edited (as quick as in seconds in chatting). That's leading to inconsistency if we cache the original one. How to notify cache servers about the new edition of a message? That's why I think they have to query the DB every time for the latest version of a message (QUORUM).

[@never308]

Do you have a discord server? If not, I think it would be great to create one, so we can ask questions and be notified of your work.

[@quintencabo]

Love the deep dive!!!!

[@AntonyXavier-v9j]

why spin-up a worker thread instead of caching in the monolith server ? is there an advantage in doing that ?

[@stunna4498]

if they used a relation database from the start would they still face this scalling problems?

[@PhuongNguyen-gq8yq]

Why are then running into one hot problem? Isn't data replicated across multiple node? Can they/cassandra just redirect the request to others??

[@stormsake]

why not simply memcached?

[@mostafamekawy5425]

I am confused with their data services choice. Why not just use a cache layer ?

[@bluecup25]

I'm wondering that too... I'm guessing the advantage of this approach would be virtually no memory usage apart from probably some buffers and keeping track of subscribers. So no extra writes for caching, just redirect the same response to multiple clients.

[@willi1978]

Coaching seems simpler than request grouping. I would prefer that too

[@bluecup25]

@@willi1978 Yeah, but caching requires memory and performing extra writes / reads.

[@erkinalp]

They prefer a dumb database, predefined queries architecture.

[@aramikm]

Caching requires invalidating! I assume based on their specific usecase that a lot of requests came at exactly same time (which was causing the hotspots) they preferred a more transient approach.

[@Bukalemur]

Does anyone knows what is the equivalent feature/plugin of the lookup feature at 7:29 for Firefox ???

[@D9ID9I]

Seems they been running for multiple years without having any caching on top of traditional hdd's with improperly working load balancer and still managed to success. Wow, modern technologies are so tolerant to weak decisions. But they still believe that Rust is the cure 😂

[@ConAim]

Now let go back to MongoDB... :)

[@lucianopanizza650]

Oh the irony

[@cybermindable]

I really like this format of video. You reading complex technical stuff and thinking about it out loud. Learned a lot and looking forward for more content!

[@Att4ni]

a little difficult to swallow but i agree that it is very beneficial, as long as you have an hour and some patience. love his videos, and he has a great presence too!

[@harriehausenman8623]

The material itself is quite complicated and as usual, all the nifty little details count, when it comes down to performance and scaling. The way it is present, this discourse (no pun intended), is very easy to follow for me and it feels like I get more understanding out of it that a lot of these expensive "certification" courses. 😆

[@Att4ni]

@@harriehausenman8623 Absolutely, I love the conversational nature of his videos. Blows my mind sometimes that people think you need to spend thousands on courses, when most information is online for free. Often times the only thing holding people back is their own will to learn

[@dhananjayraut]

I love how tagging @everyone was resulting in their on-call team getting paged for hot partition issues lol

[@SahilP2648]

Can I just say something? In today's age of disappointingly simplistic and drab logos, Cassandra's logo seems to literally be an eye of a sci-fi goddess which can see the universe or star systems. Really cool. One of my favorites, if not my most favorite logo ever.

[@emonymph6911]

Thanks for that it made me appreciate it more.

[@ValentinBaca]

(If you or others aren't aware) Cassandra was an oracle (ha! get it!) whose curse was to be able to see the future correctly but whose warnings wouldn't never be taken seriously.

[@Coding_knight]

Very informative video as usual my lord & people have commented that your video is really slow but I don't think so, many non native English speakers watch your video it's the right speed for them & at the end of the day explain the content in the speed you are comfortable in!

[@hitmusicworldwide]

True. Hussein's accent is never a problem, as are other KZbinrs whose native language is not English, and his pace is not zoomed up on mountain dew extra caffeinated speed talk, which is really unusable as way too many native speakers feel compelled to firehose out in their Gatling gun tutorials. My problem is I don't have a lot of people who are deeply into this sort of musing amongst my friends so listening to Hussein work it out with us is intellectually interesting to me and has value. I do know senior software engineers, they tell me they don't want to talk about work. They are basically 15-year-olds in 35-year-old bodies they want to talk about my specialty, international politics, but I tell them I don't want to talk about work when I'm not working as well. I'm interested in Hussein's approach because it often comes from a higher architectural level of looking at how to solve these problems I find that the most interesting thing about engineering re problem solving of all types. People like this are not just doing this to make money they're doing it because it has real value to them as an intellectual pursuit.

[@harriehausenman8623]

The faster speeds (1.5 and higher) work MUCH better when the source is slightly slower, instead of slightly faster. A lot of YTbers make that mistake (to try to talk fast) and almost everyone I know watches on faster speeds. But since everyone has their own speed, I think it is better to make the source slow, so even 2x people still get chrystal clear words.

[@notpublic7149]

Thank you for going into detail instead of the typical. 5 minutes of a ppt that really gets me no knowledge that's actually useful. You sir are pro providing a public service. TY 👍

[@24milleniums]

I have no idea what you're saying but I like your voice, so I watched for an hour.

[@harriehausenman8623]

Quality comment! 🤗

[@javaadpatel9097]

The depth of this video and the amount of knowledge shared was amazing. I read the blog post before this and learned a lot, but having you explain things in so much more detail gave me a whole lot more learnings. Thanks for the great content.

[@DJpiya1]

They could have easily rectified this with another first level cache like Redis and without implementing all those bells and whistles, including that unnecessary migration. Cassandra is always good for high throughput writes, but it is hardly recommended to use high concurrent reads by a real-time client while that writing take place. We r handling over multi Gb messages per second write with Cassandra and the problem they have mentioned aren't suprise me. The reason is, their partition key and required access pattern. They r putting all the messages related to certain discution session on a single partition, but when such a session is active all read write goes to that partition creating a hot partition. We resolve similar use cases in past by moving such highly interactive live sessions to first level cache like Redis with a TTL. With Redis we can support that message edit feature as well. This won't need a petabyte level of memory, coz we only keep the live sessions. So I don't think we need to re-invent the wheel at all. Just use the right tool for right requirement. However, may be there is a reason which is not mentioned, not to use this approach.

[@yes-ni1od]

I arrived at this video from a Twitter post I saw. This video is a lot of talking for very little reason; I feel like you are just blabbering on. I was tempted to purchase your Udemy guide, but after watching this video I feel like your guide content will just be videos of you talking about databases without any practical implementations or demos.

[@hashcheel]

Why won't they simply use a distributed cache instead of creating all that infra of data services library, worker node, managing subscriptions, etc? They introduced a bunch of failure points in the system and maintenance overhead with this solution. Isn't a distributed cache the standard obvious solution to all hot partition problems?

[@rudzon]

so they deduplicated and balanced reads

[@datasleek7950]

Yeah, not surprise about Caddandra. They should have taken a look at Singlestore DB. They store trillion of record, scale petabytes.

[@nicustroh]

cassandra will read from memory if the data is in the memtable (in memory) - otherwise it will need to use the SSTables (disk).

[@kozlovskyi]

Basically, they replaced Java garbage with same DB, but implemented in C++

[@HadiAriakia]

It is virtually impossible to make me switch notifications on for any channel on KZbin but this video left me with no choice 😂. Notifications are switched on. So informative. Love it mate.

[@sevdalink6676]

Where did you gain such an amount of knowledge and understanding? While you are explaining I can literally see how you imagine the whole micro and macro IT world. I would never be capable to understand the text alone. Thanks.

[@pollathajeeva23]

So just curious about Document db's what if we store the elastic search (Lucene indexer) with Raft consensus for consistency of data.

[@pollathajeeva23]

​@@THEROOT1111 Hmmm, Time Series runs here what do you think about this?

[@VedullaKrishna]

TBH, reading the blog post is a better choice than watching this video.

[@kevinb1594]

Whose realm would all this work/knowledge fall into? I'm a front end dev working to become a full stack dev and the complexity of all this API/backend stuff is just completely overwhelming. Considering the ever changing and wide breadth of front end technologies, I don't see how it's possible to keep up - especially since serverless and some dev ops is being pushed into our domain...

[@IvanRandomDude]

This is backend, obviously.

[@andythedishwasher1117]

We have specializations for reasons similar to the frustration you're experiencing. The trick is to find an area where you can do a lot with minimal cognitive overhead in relation to the way you personally think. You don't keep up so much as you just listen and learn patiently. And build stuff. That's important. Gotta keep building stuff or nothing will make sense in the future any more than it does now.

[@panixx8289]

What’s the difference between the data service layer and a cache, it’s basically the same thing 😕

[@tahirraza2590]

Now I can say I know a thing r two about IOs, SSDs and Cassandra

[@laksithakumara]

Discord using Scylla DB now because Cassandra is suck

[@adolfdassler7857]

what's the extension you used to look up 'chagrin'?

[@devyetii]

I think they first moved all the dbs to scylla except for the messages cluster, then started optimizing their scylla cluster alongside working on the data services, which they used for both scylla and Cassandra clusters. Their migration plan includes moving their messages cluster to scylla as well as moving their current scylla clusters to their new optimized scylla deployments. That's what came to mind reading through the article. Thanks for the useful content!

[@rocstar3000]

Amazing video and analysis. Loved it, please do more.

[@user-ij5tq6sp2x]

How do you have so much knowledge about vastly different domains when you don't even look middle aged? 😭I really wanna know. Are you so passionate about tech that you're almost always immersed in it? Do you have other hobbies? Do you get time to go out or play? Or you just a really quick learner?

[@marcello4258]

Just saw the “old” video moving to Cassandra and saw the hilarious comment about moving to cylliba😂😂

[@32zim32]

Don't understand why they can not distribute requests to unlimited number of replicas. Very strange

[@ysldev7960]

I just love the deep dive videos and discussing advanced software architecture and related topics.We should have more of these on youtube!

[@abcdef-fo1tf]

I'm a little confused about the data service serving data with request coalescing. Couldn't they of just added a traditional cache between their servers and DB?

[@kebman]

Chagrin is a special form of stress. Not really stress as in stress, but more of an annoyance, where you'd make a face.

[@filipsworks]

And all of this could be avoided by simply releasing a Self-Hosted version...

[@md-ayaz]

Do we have a similar thing for SQL database, where querying is faster and cheaper where there are billion records?

[@anuragshah3433]

Hi Hussein, loving your videos. I have a doubt and would love to discuss with you, at 51:30 you discuss how they now query the database only once by spinning up a worker node. My question is, we are still making a request to find if there is a worker running, will that not bring us back to the same problem? Or are we saying we'll be storing these worker id and the task it is working on in memory? Please do share your thoughts.

[@harriehausenman8623]

As I understand it, they first moved everything *except* cassandra-messages to ScyllaDB, then did the API/Service thing, and this bought them some time to prepare for the final migration to scylla-messages. I suppose both are "clusters" in a way. /idk confusing wording in the post.

[@paligamy93]

Shagrin is spelled with a ch???

[@myronkipa2530]

I sharted indeed

[@benlu]

I don't get why reads are reading from the database, the data should be in the memtable and multiple requests at the same time should be reading from the same memtable entry

[@Bertie_Ahern]

No idea about the subject but relaxing voice, good ASMR.

[@thirislifelogs]

This is awesome. i have gained a lot of knowledge from this. Thank you!

[@YasheshBharti]

Hussein, Love your videos :) can you do one for Vector Databases?

[@tejaswan]

Just read the article and you posted the video.

[@HowDoYouUseSpaceBar]

@@faris2448 ...wat

[@luciuspertis5672]

I didn't understand how hashing to ds would increase coalescing?? 52:52

[@richie7425]

guids can be time based guids can be sorted however which is what Cassandra can use.

[@livingdeathD]

great video👍

[@asifarko5884]

I think they migrated to scylla except one. The core I think which still was huge was in cassandra. That's where I think we have a confusion.

[@hck1bloodday]

he skipped the part in the article where it says that reverse order queryin was too slow for them, but ScyllaDB people improved that use casae for them and no longer had roadblocks to migrate the mian database, si I believe they migrated everything

[@DFPercush]

Fascinating! But why on earth can't the database cache its most recently inserted elements on its own? That makes no sense to me. Or at least cache at the OS level. It *has* to read the physical disk every time someone reads a message?! No wonder they were having latency problems. Glad to hear they solved it anyway.

[@irvinge4641]

Doing god's work omg, learned so much from just listening to you

[@user-qr4jf4tv2x]

redit allowed edit but they achieve it when post becomes idle

[@chanep1]

Excellent! keep doing more videos like this

[@CODFactory]

so what did they do with garbage collector in scylla? they are not collecting the garbage and that memory is being unused now?

[@hintzod]

scylla is written in c++. There is no garbage collection in c++, the software developer need to manually delete data in memory that is not needed, or else they will have memory leaks and dangling pointers.

[@js__k984]

great idea! Thank you for sharing

[@GoonCity777]

This video was not to my chagrin

[@hitmusicworldwide]

Soooo if I want to obfuscate and make more difficult the reconstruction of stored data, I can use random ( or perhaps seemingly random with a pattern ) uuid's. Thanks! A solution to an efficiency problem may provide a methodological attribute or layer to a cryptographic architecture.

[@EvileDik]

Indeed this is why we use randomised GUIDs for secure personal data , Discord has very low security needs so this is not so much of an issue. A typical case is where storing family data, a sequential ID scheme would add all the family members with similar values, if an attacker wants to enumerate the family members, this makes the task significantly easier, if you have an ID for one family member. As a greybeard, it is quite funny to watch new projects re-invent the wheel ore even undo hard won lessons of the past.

[@RaZziaN1]

Client messaging app ? wtf ? its not

[@AleksandarT10]

Great Blog Post! That was a great explanation Hussein! Keep up the good work

[@amarchmike]

Thanks for the good information

[@SuRFaceGoD]

Also Migration might be due the new AutoMod AI integration

[@RadityoPrasetiantoWibowo]

hi nasser, nice video !

[@shashankkumar1802]

love it. love it. love it

[@chrishabgood8900]

rust is super hard.

[@0xpatrakar]

Chagrin 😂

[@AlexBrunner94]

Will there be a Video on the recent Datadog outage?

[@MarkJones]

I wonder how long this video would have been had it recorded without the pauses in speech. Luckily YT has 1.5x playback speed

[@ebalogun1025]

Enjoyed your video 👍

[@willi1978]

cool video. Cassandra / scylladb sound interesting but it seems a lot more complex than a relational dB. How many of you tried out neondb?

[@philheathslegalteam]

Neon is good. But it has some timeout issues from S3 scale down logic as it’s in beta. For my case I had to drop it because it breaks Prisma DB pushes. They also don’t have a pricing model yet so be very wary about prod usage.

[@yxor]

Love the video, keep up the good stuff

[@yxor]

I enjoy your pace, perfect to listen to while relaxing

[@aneeinaec]

1.5x

[@harriehausenman8623]

I usually prefer the regular speed, but I discovered that the quality in even higher speeds is better, if the source is slightly slower and calmer. Some creators already speak relatively fast and then 1.5 or even 1.75 sounds pretty bad.

[@ekadet7882]

The increase playback speed button has never been so useful.

[@No_Name59]

In any case, do not communicate in Russian-language Discord servers.

[@No_Name59]

Discord disappointed. There are a bunch of characters of different sorts. There are not only characters sitting in the discord itself, there are also the persons serving the discord itself (that is, this is the technical support of the discord itself and everyone connected with it). This discord is praised from all sides. This is partly true, but there is a downside to this discord. Which few people talk about. Being in this discord can be fraught for some. Better think twice. Is it worth going to this discord? And what do you need from this discord? Pros: 1) can get caught in places, but not always a normal team for communication on different topics; 2) passing raids in games that cannot be passed without communication, even if you try; 3) designed for cooperative games; Minuses: 1) in the discord there is no "confidentiality" from the word at all .; 2) different types of characters can be caught in the discord (criminals, hackers, rich characters, humorists and characters who don't care about everything and the technical support of the discord itself), as well as young people who are not quite adequate .; 3) the attendants of the discord itself who follow the discord itself do not follow their own rules from the word at all. (Technical support and everyone who is associated with the discord). 4) in discord, only "selected" characters can scold someone, swear at someone, and so on. Otherwise, they will mix the whole discord with shit in the literal sense of the word. 5) communication with a friend on the discord is disclosed by the hackers themselves and the discord service personnel. The secrecy of correspondence is not respected at all. 6) if you start swearing with someone, then his defenders who have nothing to do with swearing will crawl out, and the persecution will start from their side, and after that absolutely the entire discord is connected. 7) hackers and "knocked down on their heads" (they can and will pretend to be) comedians begin to look for people via the Internet to annoy them at home. Seeing a discord blows their heads completely. 8) in discord, not only children, but also adults (pretending to be fools) take part in this. SO FOLLOW WHAT YOU SAY AND TO WHOM YOU SPEAK. PS: at the moment, there is no need to be present in the discord. There is a certain society that has its own laws and rules at the moment. They will mix anyone with shit. either you (the group united) with shit will mix or do not knowing who at its discretion). P.S: this is a kind of group that found its own cozy little world in discord and choose new ones and evaluate them, then they draw conclusions, and then the consequences. "Cozy little world" Here we are talking about the Russian language discord. Russian Federation (RF) Russian Discord servers problem.

[@No_Name59]

Now the Russian-speaking discord servers is a bastard habitat.discord servers role.only from the occupancy of the bastard himself.