How Discord Reduced Traffic Bandwidth by 40%

[@kikisbytes]

This video took 6,089 lines of code to bring to life. Here’s the breakdown if you’re curious: S1: 507 S2: 386 s3: 285 S4: 546 S5: 771 s6: 414 S7: 531 s8: 572 s9: 783 s10: 598 S11: 256 S12: 219 s13: 221 And it's built using Motion Canvas - motioncanvas.io/

[@faytruefireside]

What program and language are you using?

[@NiFreZ]

@@faytruefireside good question

[@unclejoe8310]

Manim

[@trashcan3958]

​@NiFreZ yeah I'm also interested

[@kikisbytes]

I use motion canvas to make my animation! motioncanvas.io/

[@MelroyvandenBerg]

Trust me, engineers don't get the bonus for this

[@HolyOllie]

youre right, they get paid 300k a year. this is literally their job. a surgeon doesnt get a bonus at every successful heart transplant

[@kikisbytes]

one can dream hehe. Exceeded performance and get some extra cash into 401k / RRSP

[@Catge]

I saved my company 30k a year in cost with some optimizations and I all got a nice job.

[@strawbaria]

@@HolyOllie Who the hell gets paid 300k unless they're staff+?

[@Sieabahlpark]

They literally get paid with stock, if discord makes more money their stock is worth more. They can then sell it when discord goes public and get paid. smh

[@Linkyboy]

"I hope the engineers get a big bonus for this" - Yeah probably, a big bonus time off to look for a new job after the layoffs.

[@kikisbytes]

that would be very upsetting :(

[@MelroyvandenBerg]

True story.

[@thegrandnil764]

@@kikisbytes "I love capitalisim" tech bros hgahahahahaha

[@Hacking-Kitten]

The bonus went to management for sure 😂

[@kikisbytes]

hahah yea....one can dream

[@namaefumei]

Its fascinating to see what a big company does to implement a new feature or a fix. So many factors, so many elements to check for. Great video. Thanks.

[@noobgam6331]

No. It's fascinating to see how people are unaware of zstd existing that was released 10 years ago and I haven't seen anyone who has every worked with logs being unaware of it and actually using gz/zlib for anything where bandwidth/compressed file size is actually important

[@kikisbytes]

thank you for watching!

[@autumnblaze6267]

@@noobgam6331 nah, I just send the full raw data, all the music is in WAV, images in BMP (hell, the background is just one color, but I don't just send CSS with background color, you get an 8k uncompressed BMP image of #ffffff, you can check every pixel individually, each one was served with love and care - hell, my engineers are working on a new system where we'll send everything as a JSON/plaintext file in the background, images will just be a giant array with all RGBA values going like [[255, 255, 255, 1], [255, 255, 255, 1]] and so on and then a giant JS script with lots of dependencies will render the images on runtime, it'll be awesome, we have the best pixels out there, it improves accessibility cos every user can just open their Dev Tools and have all the pixels laid out in a human-readable format (images aren't human-readable, they make my eyes hurt), they can read it like a novel, ofc we send all the multimedia first and only once the user loads everything (we'll test it with a checksum), we send the actual content of the website - I need to be sure users experience everything in all its glory, not loading everything ruins the user experience, the new system helps to fight all the hacks like using wget or telling the browser to just download the plain HTML file, now it'll all be sent as JSON on fly, you need to have JS enabled and actually load everything, the script keeps sending randomized unique hashes to check if you're really downloading everything and established a proper session instead of trying to just download the text, ruining your user experience in the process), all with very high bitrates and resolution - my users deserve only the best: the most pixels, the most waves, the most data! if you can't access the site, tough luck bucko, buy a better internet connection or if it's on the server due to the load, wait for another time of the day, check the downtime info, estimate based on the timezones of the users and try to access it during the few hours in a day when the servers aren't overwhelmed - the very limited exclusive access only adds to the perceived luxury, it's only accessible to the people with the best internet connection during only a few exclusive hours of the day (if you're from the US, tough luck bucko, you'll need to stay up all night and keep refreshing; in fact, I'm considering closing off the website for the Shabbat [from Friday's sunset to Saturday's sunset, according to the Israel timezone (the servers will be down) + an additional JS script blocking it according to your local timezone] to be respectful towards religious Jews), people like the feeling of being the select few who managed to access and see the website (all bots are blocked, ofc, no archiving, no cache), the exclusivity makes it precious, we're the Apple of the webdev world: only the select few, dedicated rich people, get to see it, but once you access it, you get the best experience, the website runs super smoothly cos no decompressing is happening on the user's side, everything's pre-rendered and sent as raw data, so it's rendered instantly, a super smooth user experience

[@dingus1720]

exceptional video quality, unfortunately i think i'm too orange cat brain to understand compression 💀

[@kikisbytes]

haha ty for taking the time to watch this video though! Any suggestions on how I can make it easier to understand?

[@williamine]

@@kikisbytesI didn’t find it particularly hard to understand, and I feel like you did a pretty decent job of explaining compression. I do that wish you went into more depth about what context is and how it’s used to improve the efficiency of streaming compression, though. Otherwise I thought the video was great!

[@Skaffa]

​@@kikisbytestoo many technical terms were left unexplained to non-programmers i think.

[@randoguy7488]

@@Skaffa Also there weren't enough pauses. Felt like someone was reading a wall of text without line breaks or paragraphs

[@vilian9185]

If they ban half the predators there it would also half the traffic lmao

[@Reyhz]

And half those pradators pays for nitro, they'd also halve their revenues

[@simdimdim]

They might end up banning themselves

[@vilian9185]

@@simdimdim LOL

[@ropoxdev]

So you mean everyone on Discord is predators?

[@autumnblaze6267]

@@Reyhz I'd say banning 1/2 of the predators would cut more than just 1/2 of the revenue, more like 3/4+ ya assume predators and non-predators buy Nitro at (roughly, we're all doing very vague estimations here, so it's not about the precision) the same rate, but I'd say: what normal person buys Nitro? it seems useless for normal people who just want to have a chat with friends or participate in a kinda forum every once in a while, you need to be a bit of a no-life to pay for fluff like extra emojis or whatever, it's really for the minority of users for whom Discord is their whole life AKA mostly predators, lol - now that I think about it, the profit cuts would prolly be even worse, it's probably more like "1% of predators make up 90% of the Nitro purchases", lol [I'm just joking, ofc, no hard feelings]

[@Jamesaepp]

5:48 Woooooow, almost like reducing the amount of data you send in the first place is more important than the compression you apply after the fact. Who could have guessed? Truly the most incredible engineering feat of all time.

[@kronik907]

This was my takeaway as well lol

[@wormsquirm2707]

You sound salty

[@Jamesaepp]

@@wormsquirm2707 a little. I take issue with the likes of discord taking away what was once a popular paradigm - people self hosting their VCs with software like ventrillo or team speak. Now all of this user data is going to a company - who knows how it will be used against us long term.

[@AniketSingh-nx4ds]

Man please upload more regularly if possible, just love your content, especially the breakdown of architecture and design decisions of tech companies

[@kikisbytes]

I'll try my best!! Might be a little slow with a full time job though, but definitely more to come!

[@StingSting844]

The video seems to intentionally skip fast while showing text content? You show a graph and then before even i could glance at the axes, it's gone.

[@kikisbytes]

Thank you for the feedback! I usually make the animations of how I envision the scene to look like then I try to match it with the voice recording. I find that this 2-3x my development speed. This means that sometimes the animation can be fast because the voice recording section is a lot shorter than I had anticipated. Will try my best to leave another second or two for you to absorb the graphs.

[@StingSting844]

@@kikisbytes I'm glad to know it's not intended. Great content 👍

[@Asdayasman]

@@kikisbytes I thought it was perfect, even at 2x. Honestly really good pace - usually I start a video and scroll to the commemts while the idiot talking takes 9x longer than he needs to to say anything, but this had me actually watching the video. Thank you.

[@dhillaz]

I love the quality and pace of this video - engaging explanation of the underlying concepts, use case story, test outcomes, deployment strategy...all explained in 8 minutes! Edit: Plus citations and Chapter markers in description too! 🥳

[@kikisbytes]

Thank you! Definitely a lot to cover so hopefully I was able to explain the most important concepts.

[@nessitro]

Really nice video! big bonuses -> large pizza party hahaha

[@lokylee7872]

Another great video! Keep up the good work!

[@kikisbytes]

Thank you!

[@lakshmanshankar]

Top quality content, do make more like this bro ❤

[@kikisbytes]

Thanks! Glad you enjoyed it, more coming soon!

[@arthurguedes2461]

Such an amazing Video! Thanks

[@wamellow]

really good creation! I just love such types of videos

[@kikisbytes]

Will definitely!! Thank you for tuning in!

[@Treviath]

One thing discord could do to save bandwidth would be to hash all files and checking the hash against a file database before actually sending the file to the server. Firstly it would reduce bandwidth usage and secondly it would reduce storage usage.

[@kowaihana]

the goofy ahh sound effects are too contrasting with your bored/tired voice tbh

[@kikisbytes]

Thank you for the feedback

[@farpurple]

They forkwd it to github?? ON public repo??? They are angels!

[@RupamKarmakar-s8z]

Please make more and more videos of big tech company's system design architecture 🎉🎉🎉🎉🎉

[@kikisbytes]

will do!

[@radhithyarkondaveeti]

Good video. But, I thought youll would talk more than the compression algorithms, aren't they only used for the files? Just a question.

[@pardal_bs]

Great video. I loved the little animations. What software do you use to make them?

[@kikisbytes]

Motion Canvas!

[@eliyahubasa9401]

Great video, thank you

[@fgardt]

2:20 the resulting huffman codes for a and c are swapped

[@kikisbytes]

ahhhh fk me. Thank you for catching this!

[@gabrielverle9469]

good content!

[@kikisbytes]

Thank you for watching!

[@graemepicataggio9814]

They got a pizza party and a nice work pal🎉

[@ayo4008]

The timing of this right after Russia ban of Discord is just perfect

[@kikisbytes]

oh actually? TBH I didn't even know that was happening.

[@dacjames]

Yeah, engineers won’t be getting bonuses for doing their job.

[@vippy40]

Good video, great hook

[@kikisbytes]

Appreciate it!

[@LawZist]

isn't all this compressions are built in already in the streaming frameworks / protocols?

[@dan2800]

Classic W switching from zlib to zstd

[@annyone3293]

By getting banned in Russia, haha!

[@MightyElemental]

It's a shame discord is crap now, especially the android app

[@eyepatchonmyear1318]

I get so many glitches during voice calls it's ridiculous. UI buggy as hell

[@RupamKarmakar-s8z]

please make more and more, big tech company videos

[@nomadshiba]

6:06 yeah i realized that years before. i was like, "why?"

[@skylark.kraken]

0:08 costs per what? Week? Day? Hour? It's not cumulative, it's not monthly, the points don't line up with months and looks like it's every 2 weeks but that's a weird unit of time

[@feritejakusuma8450]

i'm here till the end although i don't understand a single thing 🙃😅

[@protogen_1414]

HEEELLL YEAHH got me rolling for some reason

[@AdamBrown-dr7vd]

they reduced traffic by making the app shitty 👍

[@simdimdim]

They did it by failing to send 40% of my messages lately

[@kikisbytes]

ahahahah I had a good laugh at this!

[@lightning_11]

Wait, people still optimize their electron apps? Truly a company of cuture.

[@Stefan-qk8sw]

tldr: They switched libraries and removed unnecessary payload.

[@waste-of-time2584]

real clap. excellent

[@kikisbytes]

Thank you!

[@yofukashino_]

5:18 company of culture with no linux support ;-;

[@alexandrosang21]

I'm not smart enough to be here 😄

[@kikisbytes]

Well, your curiosity already proves you're more than smart enough to be here! 😊

[@Cyclically]

[@WillFuI]

This is cool, but I’m still upset. I can’t upload 25 MB files anymore.

[@TracyGal]

40% reduction in traffic huh? Can I have my 25mb upload limit back discord?

[@45545videos]

Common zstd W

[@vlc-cosplayer]

They banned PDF files? 🤔

[@hill2750]

I wish I could improve by 40%

[@MAXderskibidirizzler]

Interesting

[@RamkrishanYT]

Neetcode pog

[@kikisbytes]

hahha love him!

[@iohin]

6:17 will save you guys some time watching the first 6 min: they were sending useless crap to clients. They then stopped.

[@kikisbytes]

hahha maybe this should be a short :p

[@weshuiz1325]

How to reduce bandwith: Ban more users

[@klazzera]

discord lost all of its turkish users that can also skew data

[@alimahdi1012]

2:21 I bet

[@OhhCrapGuy]

1:35 That's not exactly how it would compress, interestingly. The more efficient way is A B C (3, 6, EOF) This looks like "Write A, then B, then C. For the next 6 characters, copy the character that's 3 characters back from the position you're writing." (0 indexed) When you write the [3] letter, you're copying whatever letter is 3 positions back, [0]. Then for [4], [1] is 3 positions back. [5] is [2], and this is key: [6] is copied from [3], data that has not been written when we started copying!

[@Dactarium]

Bc Discord banned in Turkey and Russia 😢

[@Ktoyatakoiskazhimne]

They could reduce traffic bandwidth by even further with p2p networking. Just send CRC of an image, and then force clients to find the image by themselves with a timeout, if there's no image among the other peers, send one from server. So in a large servers, the more people saw some image, the more you have peers for this image

[@gibarel]

Some day discord will managed to reduce their bandwidth usage by 100% by making their clients so bloated and with every feature behind a paywall so everyone will just use something else.

[@honkhonk8009]

So thats why my discord randomly disconnects and is pure TRASH lately

[@kikisbytes]

oh is that a consistent issue? Haven't experienced any disconnects

[@jaysonp9426]

Anyone complaining about the engineers not getting their share... Go start your own company or accept that you're owned by someone else.

[@yunis5097]

Sure that’s how we solve problems, everybody should have their own company. That’s a great Idea, everybody should have their own country too, so they can make their own laws. At that point just have your own planet. That’s a stupid way to think. Just accepting obvious injustices and not unionising is the first mistake.

[@RupamKarmakar-s8z]

Please make more and more videos of big tech company's system design architecture 🎉🎉🎉🎉🎉