HOW DISCORD INDEXES BILLIONS OF MESSAGES? | DISCORD | SYSTEM DESIGN | Techie007

[@harshmangalamverma]

subscribed. great content!!

[@k.k.gayansanjeewa7432]

Nice Content !. always there are new techniques to learn which never ends.

[@Techie007]

Thank you so much

[@AmanSharma-vb5jl]

you taught it like a story amazing ,complete imagination got created in brain ,thanks bro

[@Techie007]

Glad you liked it.

[@mpboardupdates]

Great Explanation🤞🤞🤞

[@Techie007]

Thank you.

[@astrix8812]

Just yesterday I was trying to find some old messages(from 2017) on an old discord server and I was blown away by the fact that discord was able to fetch those old messages so quickly which made me think about this particular topic and today I received your video notification! Thank you very much for sharing this, it was very informational!

[@astrix8812]

Also, although I like how you easily explain things, I have a small suggestion. Please do add more diagrams or small animation kind of thing if possible.

[@Techie007]

Thank you so much. Glad you found it useful. Will surely try to add more diagrams and animations

[@astrix8812]

@@Techie007 That awesome! Thank you very much

[@ujjawal_]

way you explained is amazing. I haven't read system design yet but understood it.

[@Techie007]

Thanks Ujjawal. That's great to hear.

[@mangaRoxo]

Great info

[@Techie007]

Glad it was helpful!

[@maheshrajput1703]

Thank you for awesome content! 👏

[@utkarshahunungare9452]

Informative video👍Great learning today

[@Techie007]

Glad to hear that

[@yashswarnkar1702]

Thanks for this. One suggestion when you were introducing the new concepts[ES, discord shards, lucene] it didn't actually connect well with the overall arch due to the info overload. One way would be to present the overall arch first and then deep dive into components. Something that you did while summarising. Follow up question: What happens when we have to store messages for years and allow search over it? Do we use hot-warm clusters?

[@Techie007]

Thanks for the suggestion Yash. I will definitely keep this in mind. I think Hot-Warm clusters is not for this use-case. This can be used: www.elastic.co/guide/en/elasticsearch/reference/current/rollup-understanding-groups.html

[@gokulkrishnan7427]

Great content 🔥

[@Techie007]

Thank you

[@SuryanshuDinkar]

very very informative

[@Techie007]

Thank you

[@namansingh3763]

Great Video!

[@Techie007]

Thanks!

[@ZoySauce]

Amazing content as always! Thanks for teaching us in such a fun way! 😇

[@Techie007]

Thank you so much. Means a lot

[@destroyerYT-d4s]

really valuable content

[@Techie007]

Thank you

[@logitechmr7762]

Hi @Technie007, I love the note of steps that you wrote on linkedin. could you please add these note steps on the description(show more). thanks a lot!!! ❤

[@Techie007]

Sure. I will add them.

[@shwetashukla6297]

Could it store keywords to be mapped with messageId only, why timestamp is required, just wanted to know if messageId would be sufficient or not?

[@Techie007]

Timestamp helps to query range of messages. Like Get me messages in a particular date range. This helps to create a paginated response.

[@HatedAlways]

Why doesn't Discord's blocking feature hide channel messages?

[@Techie007]

Would have to deep dive on it.

[@mahanteshambali]

Can you please point to an article if there is any.

[@Techie007]

Official article: discord.com/blog/how-discord-indexes-billions-of-messages My article: sukhadanand.medium.com/how-discord-indexes-billions-of-messages-f242e605e47c

[@mahanteshambali]

@@Techie007 thank you very much. 👍

[@vicky1035]

Great video. I guess discord recently switched to scylla db instead of cassandra

[@Techie007]

Thanks for sharing. Will check it out.

[@kongzilla2897]

When I type something on search bar the suggested messages(drop down) will change(dynamically) while I type... So my doubt is when is the request to elastic search done? Does it go for every character I type? If so then the keys must be not just the words but also their prefixes ( Like for word "drinking" I get suggestion for all "dri" or "drink" or "dr" ) ..So so much memory will be used to make the mappings even for a small message? Btw it was a great video :)

[@Techie007]

Hi, it does not make a call when you are typing. The suggestions you get while you are typing are type suggestions which you get on all websites and that is handled differently. The call is made when you press enter and it shows searching on the top and then brings all the messages with the word which you searched as highlighted.

[@saurabhsharma3009]

search engines do provide the flexibility to index all ngrams of a token to facilitate prefix searches . That feature can be utilised while indexing the data.

[@somakkamos]

I didn’t understand the storing of message id piece…if someone cn kindly shed some light So if text is cst is drinking water…. Es saves it as cat =message id, is=messageid,drinking=messageid.. Isnt that the same case as saving the whole text..??? How is data savings achieved.

[@Techie007]

"Cat is drinking the water" - storing complete sentence multiple times takes more memory than storing id multiple times.

[@nsandeep7574]

How are messages encrypted when the sentences are able to been broken down?

[@Techie007]

You can do both - Break Down the Message to generate keys And store the final message - the value as encrypted. When there is a query - reach the value - final message - decrypt it - give it to the user.

[@tyrellwreleck4226]

Cassandra was created by facebook and they released it for free. Insane.

[@Techie007]

True