How Discord Stores Trillions of Messages | Deep Dive

Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
database.husseinnasser.com
Discord engineering goes into details of how they migrated from Cassandra to ScyllaDB, improved the performance of their reads and writes and rearchitected their backend to support the new load. It is an interesting episode lets get into it
0:00 Intro
1:50 Relational vs Distributed
7:00 The Cassandra Troubles
11:00 SnowFlake vs UUID
14:30 B+Tree
19:20 B+Tree and SSDs
25:30 LSM Trees
31:00 Hot partitions
36:00 Cassandra Garbage Collector Pauses
40:00 Changing the Architecture
45:00 The Data Services
55:00 The Migration
1:02:00 Zoned Named Spaces
1:04:00 Summary
Article here How Discord Stores Trillions of Messages
/ discord
Fundamentals of Backend Engineering Design patterns udemy course (link redirects to udemy with coupon)
backend.husseinnasser.com
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
network.husseinnasser.com
Follow me on Medium
/ membership
Introduction to NGINX (link redirects to udemy with coupon)
nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
python.husseinnasser.com
Become a Member on KZbin
/ @hnasr
Buy me a coffee if you liked this
www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
/ @husseinnasser
🔥 Members Only Content
• Members-only videos
🏭 Backend Engineering Videos in Order
backend.husseinnasser.com
💾 Database Engineering Videos
• Database Engineering
🎙️Listen to the Backend Engineering Podcast
husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
partner.canva.com/c/2766475/6...
Stay Awesome,
Hussein

[@hnasr]

Fundamentals of Database Engineering Course database.husseinnasser.com

[@harriehausenman8623]

Just one small note: I think you should really like (♥) some comments, old al-Khwarizmi really seems to like that 😉

[@HadiAriakia]

Sorry mate, already purchased 😂 virtually on the release date. I definitely recommend it to anyone interested to be a good data engineer.

[@cybermindable]

I really like this format of video. You reading complex technical stuff and thinking about it out loud. Learned a lot and looking forward for more content!

[@Att4ni]

a little difficult to swallow but i agree that it is very beneficial, as long as you have an hour and some patience. love his videos, and he has a great presence too!

[@harriehausenman8623]

The material itself is quite complicated and as usual, all the nifty little details count, when it comes down to performance and scaling. The way it is present, this discourse (no pun intended), is very easy to follow for me and it feels like I get more understanding out of it that a lot of these expensive "certification" courses. 😆

[@Att4ni]

@@harriehausenman8623 Absolutely, I love the conversational nature of his videos. Blows my mind sometimes that people think you need to spend thousands on courses, when most information is online for free. Often times the only thing holding people back is their own will to learn

[@D3FKONMusik123]

The best youtuber to watch at 1.5x speed

[@fleap]

🤣

[@thedankest1974]

This guy talks way too slow. Like get to the point bro

[@brucewayne2480]

​@@thedankest1974 this is my problem with hussein , even though his content is very interesting but I wait a lot to hear something new and his videos are so long 😛

[@AbhinavKulshreshtha]

At 1.5x, over 80% of the video felt like a normal speed video. You won't miss anything.

[@alexquix6394]

I really appreciate that he talks slow, I am not native speaker

[@TylerTriesTech]

This has to be one of my favorite videos on your channel. Your live reactions and excitement about this stuff is fun to watch.

[@javaadpatel9097]

The depth of this video and the amount of knowledge shared was amazing. I read the blog post before this and learned a lot, but having you explain things in so much more detail gave me a whole lot more learnings. Thanks for the great content.

[@thatguyadarsh]

What a great in depth thought provoking awesome analysis! Thanks for all your work Hussein. Much appreciated!!

[@dhananjayraut]

I love how tagging @everyone was resulting in their on-call team getting paged for hot partition issues lol

[@HadiAriakia]

It is virtually impossible to make me switch notifications on for any channel on KZbin but this video left me with no choice 😂. Notifications are switched on. So informative. Love it mate.

[@ysldev7960]

I just love the deep dive videos and discussing advanced software architecture and related topics.We should have more of these on youtube!

[@Coding_knight]

Very informative video as usual my lord & people have commented that your video is really slow but I don't think so, many non native English speakers watch your video it's the right speed for them & at the end of the day explain the content in the speed you are comfortable in!

[@hitmusicworldwide]

True. Hussein's accent is never a problem, as are other KZbinrs whose native language is not English, and his pace is not zoomed up on mountain dew extra caffeinated speed talk, which is really unusable as way too many native speakers feel compelled to firehose out in their Gatling gun tutorials. My problem is I don't have a lot of people who are deeply into this sort of musing amongst my friends so listening to Hussein work it out with us is intellectually interesting to me and has value. I do know senior software engineers, they tell me they don't want to talk about work. They are basically 15-year-olds in 35-year-old bodies they want to talk about my specialty, international politics, but I tell them I don't want to talk about work when I'm not working as well. I'm interested in Hussein's approach because it often comes from a higher architectural level of looking at how to solve these problems I find that the most interesting thing about engineering re problem solving of all types. People like this are not just doing this to make money they're doing it because it has real value to them as an intellectual pursuit.

[@harriehausenman8623]

The faster speeds (1.5 and higher) work MUCH better when the source is slightly slower, instead of slightly faster. A lot of YTbers make that mistake (to try to talk fast) and almost everyone I know watches on faster speeds. But since everyone has their own speed, I think it is better to make the source slow, so even 2x people still get chrystal clear words.

[@AleksandarT10]

Great Blog Post! That was a great explanation Hussein! Keep up the good work

[@harriehausenman8623]

Amazing content! I like the setting, the voice is nice and calm , you take your time to think inbetween sentences (rare skill!! 😆) and the general awareness of speech is crucial for computer science (and handywork). Thanks so much! 🤗 Oh, and the actual content/deep-dive is on-point.

[@kooperl]

Blows my mind that this stuff is just on the internet for free (especially the blog) (especially Hussein's analysis)

[@nishantgoel769]

Superlike.... Your way of narrating the article makes it like a Nolan Movie. love the way you deliver this article. Thanks Hussein for creating such wonderful videos

[@davidtheprogrammer]

This is fantastic! Thanks for the deep dive, learned a lot

[@notpublic7149]

Thank you for going into detail instead of the typical. 5 minutes of a ppt that really gets me no knowledge that's actually useful. You sir are pro providing a public service. TY 👍

[@AkashKamal1998]

The discord series of yours taught me alot. Thank you so much.

[@SahilP2648]

Can I just say something? In today's age of disappointingly simplistic and drab logos, Cassandra's logo seems to literally be an eye of a sci-fi goddess which can see the universe or star systems. Really cool. One of my favorites, if not my most favorite logo ever.

[@emonymph6911]

Thanks for that it made me appreciate it more.

[@ValentinBaca]

(If you or others aren't aware) Cassandra was an oracle (ha! get it!) whose curse was to be able to see the future correctly but whose warnings wouldn't never be taken seriously.

[@sevdalink6676]

Where did you gain such an amount of knowledge and understanding? While you are explaining I can literally see how you imagine the whole micro and macro IT world. I would never be capable to understand the text alone. Thanks.

[@rocstar3000]

Amazing video and analysis. Loved it, please do more.

[@devyetii]

I think they first moved all the dbs to scylla except for the messages cluster, then started optimizing their scylla cluster alongside working on the data services, which they used for both scylla and Cassandra clusters. Their migration plan includes moving their messages cluster to scylla as well as moving their current scylla clusters to their new optimized scylla deployments. That's what came to mind reading through the article. Thanks for the useful content!

[@thirislifelogs]

This is awesome. i have gained a lot of knowledge from this. Thank you!

[@TheNayanava]

Amazing stuff!! I think memtables are stored in the form of RB trees or AVL trees, so they are already sorted, they are then serialized and stored in the form of sorted string tables. Second point additionally on compaction, it is not very efficient to perform compactions because it steals CPU cycles, and would pre-empt serving actual user requests.

[@Bertie_Ahern]

No idea about the subject but relaxing voice, good ASMR.

[@quintencabo]

Love the deep dive!!!!

[@sj82516]

Awesome deep dive into the article. I learnt a lot more comparing to read the article alone. I bought your courses and look forward to learn more on DB. I got one question I would like to ask. about the data migration, I think the dual writing is quite tricky. It cannot guarantee the data consistency. I am wondering how the make sure the data is exact the same in Cassandra and ScyllaDB, which means how they guarantee the dual write must fail or success all at once.

[@woolfel]

a common recommendation for Cassandra is date based UUID. This goes back to pre 1.0 release, so it's preferred over random UUID. Using Cassandra as a database for messaging system is a known anti-pattern going back to version 1.x days.

[@yxor]

Love the video, keep up the good stuff

[@yxor]

I enjoy your pace, perfect to listen to while relaxing

[@Erwin_Anderson]

great format) looking forward for more ) Thanks for the content)

[@chanep1]

Excellent! keep doing more videos like this

[@js__k984]

great idea! Thank you for sharing

[@anuragshah3433]

Hi Hussein, loving your videos. I have a doubt and would love to discuss with you, at 51:30 you discuss how they now query the database only once by spinning up a worker node. My question is, we are still making a request to find if there is a worker running, will that not bring us back to the same problem? Or are we saying we'll be storing these worker id and the task it is working on in memory? Please do share your thoughts.

[@marcello4258]

Just saw the “old” video moving to Cassandra and saw the hilarious comment about moving to cylliba😂😂

[@fernandozago]

This is a great use case for testing any actor model systems out there for "caching" and "coalesing" calls... =)

[@amarchmike]

Thanks for the good information

[@RadityoPrasetiantoWibowo]

hi nasser, nice video !

[@YasheshBharti]

Hussein, Love your videos :) can you do one for Vector Databases?

[@ebalogun1025]

Enjoyed your video 👍

[@0xc0ffee_]

Can I pay you for lessons/coaching? You're the most amazing teacher I've ever listened to. :O

[@adolfdassler7857]

what's the extension you used to look up 'chagrin'?

[@fs811523]

Although I had the same question about why not use cache? I think a message can be edited (as quick as in seconds in chatting). That's leading to inconsistency if we cache the original one. How to notify cache servers about the new edition of a message? That's why I think they have to query the DB every time for the latest version of a message (QUORUM).

[@never308]

Do you have a discord server? If not, I think it would be great to create one, so we can ask questions and be notified of your work.

[@shashankkumar1802]

love it. love it. love it

[@harriehausenman8623]

As I understand it, they first moved everything *except* cassandra-messages to ScyllaDB, then did the API/Service thing, and this bought them some time to prepare for the final migration to scylla-messages. I suppose both are "clusters" in a way. /idk confusing wording in the post.

[@pollathajeeva23]

So just curious about Document db's what if we store the elastic search (Lucene indexer) with Raft consensus for consistency of data.

[@pollathajeeva23]

​@@THEROOT1111 Hmmm, Time Series runs here what do you think about this?

[@livingdeathD]

great video👍

[@woolfel]

basically discord implemented smart driver to fix the issue. One long standing issue with the original thrift drivers is it wasn't smart. That caused a lot of IO thrashing and especially under load if there are hot partitions. When datastax introduced a newer protocol for non-thrift drivers, it was primarily to fix async read/write issues.

[@mystic_monk55]

Thanks for the teaching sir 🙏, keep up the good work 🙂

[@md-ayaz]

Do we have a similar thing for SQL database, where querying is faster and cheaper where there are billion records?

[@Bukalemur]

Does anyone knows what is the equivalent feature/plugin of the lookup feature at 7:29 for Firefox ???

[@24milleniums]

I have no idea what you're saying but I like your voice, so I watched for an hour.

[@harriehausenman8623]

Quality comment! 🤗

[@DJpiya1]

They could have easily rectified this with another first level cache like Redis and without implementing all those bells and whistles, including that unnecessary migration. Cassandra is always good for high throughput writes, but it is hardly recommended to use high concurrent reads by a real-time client while that writing take place. We r handling over multi Gb messages per second write with Cassandra and the problem they have mentioned aren't suprise me. The reason is, their partition key and required access pattern. They r putting all the messages related to certain discution session on a single partition, but when such a session is active all read write goes to that partition creating a hot partition. We resolve similar use cases in past by moving such highly interactive live sessions to first level cache like Redis with a TTL. With Redis we can support that message edit feature as well. This won't need a petabyte level of memory, coz we only keep the live sessions. So I don't think we need to re-invent the wheel at all. Just use the right tool for right requirement. However, may be there is a reason which is not mentioned, not to use this approach.

[@PhuongNguyen-gq8yq]

Why are then running into one hot problem? Isn't data replicated across multiple node? Can they/cassandra just redirect the request to others??

[@abcdef-fo1tf]

I'm a little confused about the data service serving data with request coalescing. Couldn't they of just added a traditional cache between their servers and DB?

[@stunna4498]

if they used a relation database from the start would they still face this scalling problems?

[@user-qd3jm9vp4y]

why spin-up a worker thread instead of caching in the monolith server ? is there an advantage in doing that ?

[@datasleek7950]

Yeah, not surprise about Caddandra. They should have taken a look at Singlestore DB. They store trillion of record, scale petabytes.

[@panixx8289]

What’s the difference between the data service layer and a cache, it’s basically the same thing 😕

[@hitmusicworldwide]

Soooo if I want to obfuscate and make more difficult the reconstruction of stored data, I can use random ( or perhaps seemingly random with a pattern ) uuid's. Thanks! A solution to an efficiency problem may provide a methodological attribute or layer to a cryptographic architecture.

[@EvileDik]

Indeed this is why we use randomised GUIDs for secure personal data , Discord has very low security needs so this is not so much of an issue. A typical case is where storing family data, a sequential ID scheme would add all the family members with similar values, if an attacker wants to enumerate the family members, this makes the task significantly easier, if you have an ID for one family member. As a greybeard, it is quite funny to watch new projects re-invent the wheel ore even undo hard won lessons of the past.

[@willi1978]

cool video. Cassandra / scylladb sound interesting but it seems a lot more complex than a relational dB. How many of you tried out neondb?

[@philheathslegalteam]

Neon is good. But it has some timeout issues from S3 scale down logic as it’s in beta. For my case I had to drop it because it breaks Prisma DB pushes. They also don’t have a pricing model yet so be very wary about prod usage.

[@kevinb1594]

Whose realm would all this work/knowledge fall into? I'm a front end dev working to become a full stack dev and the complexity of all this API/backend stuff is just completely overwhelming. Considering the ever changing and wide breadth of front end technologies, I don't see how it's possible to keep up - especially since serverless and some dev ops is being pushed into our domain...

[@IvanRandomDude]

This is backend, obviously.

[@andythedishwasher1117]

We have specializations for reasons similar to the frustration you're experiencing. The trick is to find an area where you can do a lot with minimal cognitive overhead in relation to the way you personally think. You don't keep up so much as you just listen and learn patiently. And build stuff. That's important. Gotta keep building stuff or nothing will make sense in the future any more than it does now.

[@willi1978]

I guess they could use Scylla/Cassandra in analogy since Scylla is a rewrite of Cassandra that claims to be 10x faster

[@xslvrxslwt]

because it is. everything that uses Java is so bad 💀

[@tejaswan]

Just read the article and you posted the video.

[@HowDoYouUseSpaceBar]

@@faris2448 ...wat

[@nicustroh]

cassandra will read from memory if the data is in the memtable (in memory) - otherwise it will need to use the SSTables (disk).

[@AlexBrunner94]

Will there be a Video on the recent Datadog outage?

[@luciuspertis5672]

I didn't understand how hashing to ds would increase coalescing?? 52:52

[@CODFactory]

so what did they do with garbage collector in scylla? they are not collecting the garbage and that memory is being unused now?

[@hintzod]

scylla is written in c++. There is no garbage collection in c++, the software developer need to manually delete data in memory that is not needed, or else they will have memory leaks and dangling pointers.

[@richie7425]

guids can be time based guids can be sorted however which is what Cassandra can use.

[@kebman]

Chagrin is a special form of stress. Not really stress as in stress, but more of an annoyance, where you'd make a face.

[@VedullaKrishna]

TBH, reading the blog post is a better choice than watching this video.

[@tahirraza2590]

Now I can say I know a thing r two about IOs, SSDs and Cassandra

[@iyxan2340]

I really love the deep dive format of these videos, I just wanted to give a bit of a suggestion: can you talk a bit faster and like get rid of delays? I occassionally get bored being too impatient about what you're going to be explaining. It's a bit of a shame for me that these very interesting content becomes boring just because of the way you talk. Anyways, keep up the good work 👍

[@user-qr4jf4tv2x]

redit allowed edit but they achieve it when post becomes idle

[@DFPercush]

Fascinating! But why on earth can't the database cache its most recently inserted elements on its own? That makes no sense to me. Or at least cache at the OS level. It *has* to read the physical disk every time someone reads a message?! No wonder they were having latency problems. Glad to hear they solved it anyway.

[@irvinge4641]

Doing god's work omg, learned so much from just listening to you

[@benlu]

I don't get why reads are reading from the database, the data should be in the memtable and multiple requests at the same time should be reading from the same memtable entry

[@kozlovskyi]

Basically, they replaced Java garbage with same DB, but implemented in C++

[@mostafamekawy5425]

I am confused with their data services choice. Why not just use a cache layer ?

[@bluecup25]

I'm wondering that too... I'm guessing the advantage of this approach would be virtually no memory usage apart from probably some buffers and keeping track of subscribers. So no extra writes for caching, just redirect the same response to multiple clients.

[@willi1978]

Coaching seems simpler than request grouping. I would prefer that too

[@bluecup25]

@@willi1978 Yeah, but caching requires memory and performing extra writes / reads.

[@erkinalp]

They prefer a dumb database, predefined queries architecture.

[@aramikm]

Caching requires invalidating! I assume based on their specific usecase that a lot of requests came at exactly same time (which was causing the hotspots) they preferred a more transient approach.

[@SuRFaceGoD]

Also Migration might be due the new AutoMod AI integration

[@hashcheel]

Why won't they simply use a distributed cache instead of creating all that infra of data services library, worker node, managing subscriptions, etc? They introduced a bunch of failure points in the system and maintenance overhead with this solution. Isn't a distributed cache the standard obvious solution to all hot partition problems?

[@woolfel]

The old RDBMS master-slave design doesn't scale well for global distributed platform. Even if you use transactor design like Datomic, you can still overwhelm the cluster. Then there's designs used by data grids like Coherence. Scaling a database to trillions of message is tough. Even oracle RAC would have a tough time scaling for this type of load.

[@asifarko5884]

I think they migrated to scylla except one. The core I think which still was huge was in cassandra. That's where I think we have a confusion.

[@hck1bloodday]

he skipped the part in the article where it says that reverse order queryin was too slow for them, but ScyllaDB people improved that use casae for them and no longer had roadblocks to migrate the mian database, si I believe they migrated everything

[@catharsis222]

This video was not to my chagrin

[@paligamy93]

Shagrin is spelled with a ch???

[@32zim32]

Don't understand why they can not distribute requests to unlimited number of replicas. Very strange

[@stormsake]

why not simply memcached?

[@rudzon]

so they deduplicated and balanced reads

[@MarkJones]

I wonder how long this video would have been had it recorded without the pauses in speech. Luckily YT has 1.5x playback speed

[@chrishabgood8900]

rust is super hard.

[@0xpatrakar]

Chagrin 😂

[@ConAim]

Now let go back to MongoDB... :)

[@user-ij5tq6sp2x]

How do you have so much knowledge about vastly different domains when you don't even look middle aged? 😭I really wanna know. Are you so passionate about tech that you're almost always immersed in it? Do you have other hobbies? Do you get time to go out or play? Or you just a really quick learner?

[@myronkipa2530]

I sharted indeed

[@RaZziaN1]

Client messaging app ? wtf ? its not

[@ihateidiots9484]

The problem exists in entire IT industry. Facebook created React for Facebook needs and lots of stupid developers start to use it because "it's a Facebook way, they can't be wrong". But most of apps have different requirements and are not social network at all. That leads to awful architecture and stupid design. The same problem exists in the backend side. I've seen a lot of stupid decisions in projects just because they are the "Netflix way of making microservices". Facepalm.

[@filipsworks]

And all of this could be avoided by simply releasing a Self-Hosted version...

[@D9ID9I]

Seems they been running for multiple years without having any caching on top of traditional hdd's with improperly working load balancer and still managed to success. Wow, modern technologies are so tolerant to weak decisions. But they still believe that Rust is the cure 😂

[@ekadet7882]

The increase playback speed button has never been so useful.

[@laksithakumara]

Discord using Scylla DB now because Cassandra is suck

[@yes-ni1od]

I arrived at this video from a Twitter post I saw. This video is a lot of talking for very little reason; I feel like you are just blabbering on. I was tempted to purchase your Udemy guide, but after watching this video I feel like your guide content will just be videos of you talking about databases without any practical implementations or demos.