How Discord Stores TRILLIONS of Messages

Have you ever wondered how Discord is able to store trillions of user messages without breaking a sweat? In this video, we'll take a deep dive into the technologies and strategies that Discord's software engineers use to manage such a massive amount of data.
SOURCE: / discord
Discord is one of the most popular communication apps in the world, with over 250 million users and counting. With such a large user base, Discord's servers receive an enormous amount of messages every day. To store all these messages, Discord employs a distributed database system that runs on hundreds of servers worldwide.
This system uses a variety of modern database technologies, including Cassandra, Redis, Elasticsearch and most recently: ScyllaDB to ensure that messages are stored reliably and efficiently. These databases are designed to handle huge amounts of data and provide fast read and write capabilities, making it possible for Discord to scale to millions of users without compromising performance.
So if you've ever wondered how your messages are stored on Discord, or if you're just curious about how modern database technology works, this video is a must-watch. Join us as we explore the world of distributed databases and learn how Discord's software engineers keep trillions of messages organized and accessible for millions of users around the world.
👻 MY NEW STARTUP: www.caspr.ai/
MY NEWSLETTER 💌
thebetter.dev
------
CONNECT WITH ME ON SOCIAL
📸 Instagram:
/ lewismenelaws
🎚TikTok:
/ lewismenelaws
🐣 Twitter:
/ lewismenelaws
--
My gear 💻
liinks.co/lewismenelaws
-----
TIMESTAMPS ⏰
0:00 Intro
0:38 The BackStory
1:46 Cassandra Troubles
4:09 Change Architectures
6:28 Data Services
8:57 The BIG Migration
10:35 Where It's at now
11:40 What we can learn from this
12:46 Thanks for watching and subscribe :)

[@CodingWithLewis]

Thanks for watching this video :) What company would you like me to cover next?

[@infinitelancer4612]

Reddit

[@ExplodeCode]

Do How ExplodeCode is cool. naa do snapchat or instagram or something based with pictures

[@adill6295]

Rust roadmap

[@akvamsikrishna5535]

How telegram makes money and how it stores as much as data like discord and much more (you too will get more questions once you start to think how telegram is able to do it 😅)

[@akvamsikrishna5535]

And 😅 I want you and forrest knight to have a video together 🥺

[@sorteslyngel2k]

How the hell is discord even making money. That infrastructure sounds pretty expensive...

[@agentnull5242]

Nitro 😭

[@agentnull5242]

Bloody expensive too

[@lucachiapello6454]

Data

[@tonik2558]

Venture capital is one hell of a drug

[@MtTheToto]

They are not making money. Welcome to capitalism

[@jamoncitovideos]

This types of videos are great. It's not common to face scenarios that have this massive scale and it's great to get to know how they have been handled before by other engineers

[@puspamadak]

The engineering was truly awesome. Have to salute the engineers as well as SyllaDB team for such a great product.

[@ryojikn]

This is amazing, please keep going. Such an amazing source of knowledge and inspiration in a perfect method for better engineers

[@5095smf]

Nice! I love databases and learning more about them, great video.

[@coolxify]

😂 same

[@staticobjx7232]

Hi Lewis, I discovered your channel thru YT Shorts, but I like this format a little better, not only it dives into architectural problems and solutions, in my opinion it is not rushed. Also, awesome graphics! Thanks a lot and more power to your channel.

[@nerduser1000]

The type of content I love to watch thank you lewis! Had this question for Reddit how reddit saved and would love to know how discord works

[@demin-e]

No © symbol next to “Rust”? Lawsuit is already being filled

[@wonkywonky6307]

This is fantastic! Congratulations to all those amazing engineers and congratulations to you for making this great video. Thank you for bringing this content to us.

[@vaggelis_best]

11:24 Poor Descord engineers😢 They are stressing even HARD

[@Raul-pg1pf]

lmao

[@nemzyxt]

Whoa! This is so enlightening, thanks Lewis! The engineers are doing really great work bts

[@AdoSean]

Really fascinating video and well made. Thank you!

[@walterrodriguez9550]

great breakdown and description, thank you!

[@ryanzspaulding]

I like your funny words magic man. Definitely didn’t understand about 95% of this video but it was really fascinating and well presented

[@TheMassgames]

This types of videos are great, and your animations are really on point

[@Luxcium]

I am so happy 😁 to see how it goes with your channel and how much your topic is related to more advanced topics

[@NaviYT]

I was today years old when I learned that Lewis makes long form content.

[@joshburger2217]

as a dev, just thinking about a migration of that scale is anxiety inducing

[@My.MusicChannel]

This video was perfect omg!! Liked & subscribed, keep it up ❤️❤️

[@xenofenus]

This was an incredibly well made video!

[@bvorg]

Everything is perfect. Thanks much Lewis!

[@def1nt]

Very cool format and interesting stories! I wish Discord to find frontend devs as good as their backend team

[@cocoiyeah]

Kudos to the ScyllaDB team! I've been a fan of their product since 2019, and in my opinion, it's better than Apache Cassandra. However, their open-source license limits usage to only five nodes.

[@Jecsham]

the ydeserve being paid for that product

[@dominikmankowski1636]

Not really. The 5 nodes limit applies only to ScyllaDB Manger, not the OSS database.

[@pietrodc0]

72 nodes per 140mln users = ~1.9 users per node. 5 nodes limit = ~7.72mln users = you must have money for DB licenses 😅

[@OT-tn7ci]

​@pietrodcfab but that's not how it scales

[@ThatBubbledGuy]

Your channel needs to be more popular. This content is the only thing that motivates me to write some code after running into an error.

[@justingolden21]

Love the video and also the short takeaways at the end

[@Fingiorstyle]

Thank you for this video! Please create more

[@peckychicken]

Wow discord are really committed to keeping their app up to date

[@denitechYT]

Aha so now we know how! Great video man!

[@BrendAn_RSA]

This is my first video on your channel. Loving it. Awesome video :)

[@TNTISGOOD]

agreed, my first video too! :D

[@smixqse]

i love the way the subtitles interpreted scylladb: s db, mes db, solid b, sillydb, facilitydb, silladb, sil a db

[@thewhitefalcon8539]

still a DB vs no longer a DB

[@somnathroy102]

Thank god yt recommended this. I learned a lot.

[@AnushR27]

Great video Lewis 😃. Got to know so much information about Discord databases. Thankyou!

[@Skeffles]

Great video. Fascinating to see how they solved this!

[@bobtheoutbuilder8552]

I can imagine reading discord messages from 47 years ago.

[@felipheleal]

Excellent video 👏🏼👍🏼

[@shloktadilkar5536]

Very nice video Lewis

[@nathan0401_]

This was very fascinating to watch

[@costelinha1867]

Bold of you to mention the Rust language and use it's logo in this video. Considering what the Rust Foundation has been up to recently.... AND YOU SHOW A PICTURE OF THE LOGO IN A DIFFERENT COLOR... oh my god, I hope the foundation doesn't see this.

[@CodingWithLewis]

🤣🤣🤣

[@frogery]

love the animations on these videos

[@roshangamage7749]

Great video ❤

[@nickhtt]

What a nice video. Thank you very much :)

[@JoeyClover]

The concern you described at the end is a well known anti-pattern called the Golden Hammer. Don't get too comfortable with the tools you know and use.

[@YosephTeki]

wow very very insightfull !!! thanks for making this video

[@AhmedMi1ad]

Wow I really enjoyed the video, so insightful.

[@joker-wr8pt]

Sir you're my inspiration these types are useful please bring more like this like how big system works so that we can have real-time understanding

[@MultiSciGeek]

This is better than ASMR. Fell asleep in 8 minutes!

[@bajagainsatish4828]

Awesome dude!!!

[@IgnitedIce81]

Never realized how incredible the engineering is on Discord

[@tanmaypanadi1414]

the editing on this channel has exponenetially increased in quality 🤩

[@Keys_02]

Great content, please I want to learn about the internet from a developer's perspective what book are you going to recommend for me?

[@ojasmittal8381]

This guy basically turned Discord's blog into an interesting video 10/10.

[@ninjanape]

Great video. Subbed.

[@po6577]

this story are amazing, great motivation to improve myself!

[@avonzo]

Great video. Great content. Thank you.

[@Tarodev]

Great video!

[@MistaWu]

That was cool… awesome breakdown…🔥🔥

[@varshard0]

So informative and entertaining.

[@-hackers_industry]

Could you do "A roadpath to become a software engineer" some time soon?

[@Terracronz]

Cool video, earned a sub

[@willmil1199]

Love this channel

[@Joshuahendrix]

Really enjoyed this one ☝️

[@weiSane]

I'm waiting for the day discord switches to vector databases when they have to store zetabytes of messages 😂

[@nuvotion-live]

I can see it happening. I’m sure the singularity will shake up technology a lot, and it does seem like vector databases will continue to become more popular

[@kumalalalalalalaa7045]

Love these videos

[@awsomevideoperson]

I wonder if the issue with the normal Cassandra is really the GC, or the memory pressure Java currently has due go reference types. I suspect user defined primitives and fibers will really make Java a stronger contender in the high performance space

[@farhan-momin]

what a video! learning + entertainment

[@zuman_one]

Please Keep making such system design videos because I don’t like to read long articles 😂

[@javadevelopment6551]

Great video, and I'm left with one question how to find and use the right tools at the right place and at the right time 😅 That something comes with experience or any beginning can also master that?

[@pemessh]

Wow this is an awesome informative video. Please do keep them coming!!

[@mukithasan9684]

Today i learned a lot about database management how behind the scenes work

[@allezvenga7617]

Thanks for your sharing

[@MyNameIsSalo]

have never put much thought into discord but they really are at the peak of social media, almost 0 downtime ever and always provides a consistent streamlined experience. I can imagine that they will end up doing many many more database migrations in the future as discord dominates the social media space. Wonder if there will ever be a permanent solution of a database which is truely infinitely expandable.

[@musikSkool]

Discord is the perfect combination of Twitter, Facebook, an in-game chat bar, and even Instagram. If you pay them a little you can even treat it like KZbin or Twitch.

[@broneverasked]

Lewis: *thinking he says NoSQL bad* Me: *says NoSquirrel*

[@aze4308]

Great video.

[@simonkimel3339]

This shows that Discord developers are really capable, yet the auto update on Linux doesn't work.

[@Tudas]

Because Linux is literally completely irrelevant, atleast on the consumer market.

[@oh-noe]

@@Tudas lol so dumb

[@lucass8119]

@@Tudas You would think Discord, a company that relies on 100% of their own servers running Linux and is a consumer of open-source products like Apache Cassandra, would put just a tad more care into the linux version.

[@Nitidus]

​@@TudasI mean, there's the Steam Deck now. It definitely has become a LOT less irrelevant compared to a couple of years ago.

[@umaradam3788]

​@@Tudaswhat are you talking about. The entire consumer market is reliant on Linux. Facts.

[@bideshbanerjee5506]

I may be wrong but I heard from someone in a livestream that data service layer can be implemented in golang and the latency difference would still be unnoticeable

[@rkubapl]

great video

[@skiper7795]

I love discord. Best comunicator out there

[@TomWhi]

Interesting video. I feel sorry for that underused blackboard though… it looked like it was going to be written on about 5 times with not a single mark ever made on it!

[@sebobebo10]

I read the title and immediately thought, “I wonder how much storage space I can take up by just spamming emojis…”

[@TheDingsBoms]

Inspiring!

[@DemiImp]

You just altered the color of the rust logo. That is in violation of their trademark!!! 😂

[@Vergilenjoyer]

Subtitles: "Hi, my name is, and" Video: "Hi my name is Lewis"

[@Benoit.mp4]

Very interessing !

[@god_of_everything9644]

Got urself a sub sir

[@orlovchik]

thanks for the video!!! the question was in my mind this morning and, what a great recommendation, you gave me a full answer! but I have one more question why discord mobile app is so laggy and slow, while its web and desktop versions works fine

[@vanilla-plus]

Interesting colour choice on the Coalesce... What an interesting hypothesis.

[@csabraxas]

I feel like you are trying to target newer programmers when you explain very simply concepts but then at the same time you gloss over extremely advanced concepts as if we understand them already. I'm not really learning much. I would love like a 5 part series of this whole thing in extreme detail. How does scaling work? How does spinning up clusters/nodes work, how are requests routed? How does GRPC work.

[@tsarprince]

Awesome!

[@lisibethklair]

every time I think about discord I get a horrendously awkward flashback to my first situationship :/ when he asked “do u have discord” I replied “about you staying or about you leaving?” 💀

[@Dahlah.FightMe]

Nice Bro :D

[@vncntjms]

"what any programmer would want to do when they see something new: changing architectures" I'm guilty.

[@SusGus-rf8gm]

A few years ago, I don't remember why exactly but I had issues my phone plan and I was stuck with worse than dial up speeds for a while. Im talking low kilobytes per second, maybe even bytes. Discord was literally the only "online" app that still worked pretty much normally (besides embeds). That thing is optimized to death

[@harrywang4769]

yeah that's not it. the size of the data being sent and received has no correlation between how optimized the app is. your internet connection works for discord because the amount of data sent by discord is very small. it has nothing to do with app optimization.

[@shuaibalghazali3405]

Just so amazing

[@Loading_Code]

I only watch your videos because your expression are amazing

[@Dampfaeus]

Nice keyboard!

[@toopkarcher]

Very cool

[@youtubear2217]

"2017 was 6 years ago" nooooooo😭

[@itsjustsushi2023]

This could've been the perfect video to advertise your dc

[@thewhitefalcon8539]

He's got his own data center? Nice