Solving distributed systems challenges in Rust

In this stream we work through the fly.io distributed systems challenges (fly.io/dist-sys/) in Rust, and solve all the way up to challenge 3c.
You can find the code at github.com/jonhoo/rustengan.
0:00:00 Introduction
0:05:57 Maelstrom protocol and echo challenge
0:41:34 Unique ID generation
1:00:08 Improving initialization
1:31:10 Single-node broadcast
1:50:28 Multi-node broadcast and gossip
2:49:15 Don't send all values
2:57:39 Improve efficiency of gossip
Live version with chat: kzbin.infoUgf-EgWBExs

[@user-ko4fi2sv2i]

Still the best Rust related stuff! Really appreciate what you are doing!

[@ozturkberkayy]

Please do more distributed system videos like consensus algorithms, distributed databases etc

[@ErickBuildsStuff]

I would second this motion. Consensus algorithms in rust. Also Byzantine Fault Tolerant State Machine Replication

[@vijay14october1984]

I too want the same

[@Funcijej]

I hope you do more of these, especially with diagramming. It really helped to visualize the problems you were solving.

[@rcjara]

Your content has always been fantastic, but this was my favorite video and the one I got the most out of. I think it’s because of the clear and digestibly sized problems. I don’t feel like I need to focus for a whole 3 hour block of time to fully get everything out of it. I can start and stop the video at my leisure. It also gives me a clear project I can do if I feel I want to practice what was just learned. Similar vibes to the time you worked through the rust macro book. Obviously, make the content you want to make, but you’ve definitely left me wanting more in this genre.

[@jRsqILVOY]

It'd be great to see more videos on distributed systems - like implementing RAFT and other consensus algorithms, query routing, distributed hash tables, distributed storage, etc.

[@sugasheeze]

I'd be extremely interested to see how someone like you--with your literal doctorate in distributed systems--would build up a similar challenge.

[@jitcorn1103]

I always learn many new things from your videos. Thank you so much!

[@bsodmike]

Awesome as always Jon, enjoyed this thoroughly. I appreciate that you are a master at articulating your thought process and reasoning behind various changes taken (and the pros/cons at those ultimately taken).

[@MegaPacoquinha]

Now I have something to watch through easter! Ty Jon

[@BertPdeboy]

Funny how enthusiastic you are about "Rustengan" :D I APPROVE!

[@bilyzhuang9242]

Appreciate your rust coding video. I am a beginner for the rust and I really learn really practical rust knowledge by following your coding line by line. Thank you !!! Wish you a good life.

[@beersheep0109]

Amazing. Looking forward to more similar contents.

[@maywakeTV]

Thanks for sharing! Just got interested in learning Rust again and found your book on Kindle and via that your KZbin!

[@vishalsharma-bp9zu]

Thanks for the stuff, high high quality, not sure what the 20 dislikers were expecting from this video.

[@polares8187]

It was so satisfying to see you pass the 3c without changing anything.

[@user-sn3tf7gf9r]

Thank you for this amazing walkthrough!

[@topdeveloper4794]

i hope you provide us a video which is continue to solving rest of the problems

[@user-qq9zl6lt6d]

Hey! I remember Jepsen for their Consistency Models page! It was really helpful to get me started on distributed systems and understanding data consistency, along with potentially building hand-wavy reasoning/logical system around it! Happy to see fly-io take the author far far away onto actual courses for distributed systems!

[@JustSomeAussie1]

Never took Jon for a Naruto guy

[@bsodmike]

rustengan was an epic choice!

[@bemofresh3117]

I literally asked the question “is there any material for distributed systems from the Rust perspective?” On LinkedIn, and was told that I was asking a stupid question! Can you believe that?

[@mechanicalmonk2020]

Yes. It's linked in. It's like the worst parts of Reddit and Twitter combined

[@narigoncs]

Great video as usual!

[@YTCrazytieguy]

I think your gossip optimization has a bug - the topology doesn't need to be symmetric, so a node doesn't necessarily know about messages that nodes in its neighborhood know about. Had a blast, thanks!

[@saksmlz]

Thank you so much for doing this ❤

[@LB767]

39:13 OMG the amount of times I've had this problem, and being a beginner had no clue how to fix, so I would end up trying to shuffle lines around or clone or god knows what, when dereferencing was all that's needed... This is why I love your vids, they contain so much helpful practical knowledge, they are just a joy to watch.

[@alexxi3955]

Great vid, loved it!

[@andydataguy]

Glad to see you appreciate DnD 🐉🙌🏾

[@narigoncs]

One thing that would have been interesting to see is what kind of an effect changing the gossip interval has on the latency to consistent, in combination with the 10% extra items sent per gossip!

[@floriantavares1849]

I loved this stream ! Very interesting. Have you in mind to do an other stream focusing on consensus protocol, traditional leader based (Paxos or Raft) or even new leaderless alternatives (Epaxos, Atlas, Tempo, Accord) ? That would be awesome !

[@lightninginmyhands4878]

I like that this guy looks up cool words to name his cargo

[@bsodmike]

He does this regularly and comes up with some really interesting ones. Defenestration was one he picked (I think). Do you realise how hard it is to name a lib well? "There are only two hard things in Computer Science: cache invalidation and naming things."

[@TNH91]

@@bsodmike I think you forgot off by one errors ;P

[@bsodmike]

@@TNH91that and runtime segfaults!

[@alejrandom6592]

Hi, I woke up and this video was on. Nice stuff

[@chrissaltmarsh6777]

In relation to distributed (heterogeneous, asynchronus) systems, you might be interested in Petrinets. If you are actually building a system, this gives you a tool for catching some of the problems before you design them in. It is a bit like the relationship between ER modelling and database implementaion. I'm using Rust for my 'nodes' (made from 'Place' 'Transition' and 'Arc' in PN language) and Rust fits perfectly.

[@cooldude3010]

Love this content. More of this.

[@natenatters]

Very cool video! Thanks :)

[@bjugdbjk]

Common need something like these, Bring rust to Backend stuff !!

[@ZhanhuiLi]

Very helpful, thanks。

[@ClineMusic00]

Would love to see the next challenges

[@aleksandr-belousov_1]

Man, I missed your content! :)

[@saurabhshinde1855]

Hii John, Love your work !! Hugh respect!! What is your take on decisions made by The Foundation on "Rust" trademark and logo usage ?? I felt little disappointed..

[@Lockeinfurry]

Great video as always! I think you are still seeing big Gossips in the end because you are making an assumption that all Gossiping happens bi-directionnally. If n1 Gossips to n2 but n2 never gossips to n1, n1 will always want to send the full thing as it doesn't knonw that n2 now knows the stuff it sent to it previously. That's where the GossipOk messages might come in handy so that n2 can tell n1 it saw the messages n1 Gossiped to it.

[@nachiketkanore]

Thanks for the inspiration

[@dmitrij34]

Hi Jon, amazing as always! I had an idea for sync across nodes - can you form a probabilistic data structure(like a bloom filter) from your node’s values and send it to neighbours, so they can test for known values, to figure out the data that we might be interested in? Sure, there may be false positives(we don’t know about the value, but the bloom filter says we do), but I feel like they can be dealt with. Or use a lossy hash table for inverse behaviour(only false negatives)

[@jonhoo]

It's a cool idea - worth trying out!

[@berndeckenfels]

3:01:30 instead of random you can use a module counter so you send n, n+1 and n+2 additional alreadyknowns. Or you add the first x of all already now since in the next message they won’t be contained

[@Augustine_354]

This is gold.

[@blob4000]

Thank you so much!

[@tommy_salami108]

Hey Jon, I think it would very interesting to implement eye tracking. For example, at 22:39, you tab back to the serde_json docs for just a moment, then find the answer of what you're looking for. Seeing exactly what you looked at on the page would clue us in to your thought process, and add another layer. -Tom

[@Funcijej]

That would be neat but I don’t know how easy eye tracking is to set up without dedicated hardware

[@random6434]

When you defined Body with a field rest: B and then rejected it, you could actually keep Body and use Body for two-step deserialization and Body for one step deserialization if you happen to know B should be. There's a slight bit of weirdness in the second case you end up with a redundant string but you could make a type with two fields, a hashmap of serde_json::Values and a string for the type: struct DynBody { ty: String, fields: HashMap}. But deserialization to a KnownType might need a little bit of custom code to check the type field and reject if it's not the expected type. Just some random thoughts.

[@dandreani]

39:09 when even Jon get stuck with the borrow checker... i think i am safe

[@ningzhi3415]

It's really an excellent stuff.

[@L0wPressure]

1:01:27 - hehe, Beavis, he said "generic over some ASS" I kinda understand the gist of the code, but i feel like it's still way over my level at the moment. Awesome content anyway :)

[@ruijungao]

Great work! However, just wonder why not just return a Result, and let nodes forget serialization stuffs?

[@dunghdinh6268]

Amazing !!!

[@JohnKoepi]

There is a crate that implements fully RPC/KV spec if you need any kind of support to fully focus the most interesting challenges - Kafka / Tx.

[@johannessandjaja7630]

hi, i was wondering on 39:14 in terms of ownership and borrowing is there any difference between `&mut *output` changing the function param to `... mut output: &mut StdoutLock...` and just call `... serde_json::to_writer(&mut output, &reply).context("serialize response")...` anyway great content, thanks!

[@bjugdbjk]

Ths s so cool !!

[@leviwebercosta8154]

Hey I know this is not in the topic of Rust exactly but can you post a video of your coding setup? (ide, vim scripts ...)

[@viniciusrolandcrisci272]

Amazing content, Jon! I have a little (maybe dummy) question: At the end you added an eprintln! for debug the size of messages sent. Why that operation didnt break the reading from stdout? How Maelstrom knows what is a message response and what is a log from stdout? Sorry if I missed something 😅

[@jonhoo]

Ah, so, `eprintln!` (note the `e`) prints to STDERR, not STDOUT. It's `println!` (no `e`) that prints to STDOUT. Since Maelstrom only checks STDOUT, you can safely use STDERR for other kind of printing/logging.

[@viniciusrolandcrisci272]

@@jonhoo Oh, that makes sense! Thanks! Keep up the excellent content!

[@topdeveloper4794]

Thanks. Can you reupload eng subscription version?

[@kopo88]

In the echo example, would `println!("{}", serde_json::to_string(&reply)?);` have worked instead of explicitly working with the `StdoutLock`?

[@jongxina3595]

Working on 4. Im not sure I understand but they provide a service for kv storage so to replicate this in Rust it means I would just use a database like Redis right?

[@JohnKoepi]

Nice!

[@user-ue5bx8sp6r]

Very nice content. I think it's the first time I watch a live for so long. Does anyone know someone doing similar content in go?

[@__J____ff]

whats that Panda symbol that shows up next to payload: Payload popup ? what vim plugin is this ? and whats the on the go errors from ?

[@zilianglin7417]

Hi Jon, thank you for the great streaming. At around 2:23:49 I notice there are proc macro errors after you gd into other crates. Are there ways to avoid those?

[@mishikookropiridze]

I believe no. probably changes in rust-analyzer needed.

[@seftondepledge3658]

Does anyone know if there is still a suggestions website for future streams?

[@gimmypharel]

Hi Jon, awesome stream as always! Got a question about the "improving efficiency" part. Why dont you just keep track of which messages you've already sent to the peer and stop sending those? Is it because you are not sure whether those have arrived or failed due to network conditions?

[@jonhoo]

That's exactly right!

[@guilhermesoares7857]

​@@jonhoo In this case, implementing a perfect link to mitigate this problem wouldnt benefit you in terms of performance ( not to mention that the program would have to be async ) right ? Watching you program/explain these distributed system challenges is giving me a new love for this topic!

[@mishikookropiridze]

@@guilhermesoares7857 really perfect link can never exist

[@lker-dev]

For the gossip section getting longer was there also the possibility based on provided topology that ‘n1’ talks to ‘n2’ but ‘n2’ doesn’t talk to ‘n1’? Or were they always bi directional without the need for the sync?

[@matthewpublikum3114]

What plugin is that in vim? What's a good code completion plugin?

[@Script_Alchemist]

Gosh this setup, I had nothing predownloaded. That took a while

[@Script_Alchemist]

I'm not sure what happened. I created the project like you and my rust-analyzer broke. I debuged for 3 hours. Deleted the repo. Recreated it without the --bin and it works. Not sure that happened there, but I'll write that in case it helps anyone else

[@daphenomenalz4100]

How do the other Nodes get the Gossip message? The InjectedPayload Event writes to stdout, but the Nodes are only reading from the stdin which is recorded through the channel.

[@felgenh399]

love the name

[@rawsteelfsp]

Can someone tell me what drawing tool is being used here?

[@DarkKnightUNS2008]

I only remembered Rasengan after 3 hour of video :)))))

[@Ropbastos]

What drawing program is being used?

[@ElGnomistico]

Do you have any interest in trying to implement a chess engine in Rust? I do and watching you do it would be very instructional.

[@bocckoka]

r#type? we have a lot of that.

[@tsalVlog]

Of all the streams for me to miss....

[@ShikaIE]

Would you continue this challenge in future?

[@user-bk1mj4ds4i]

(Noob Q) why are stack traces in Java ? Tyia

[@myname2462]

Best quality content at 7:30 😅

[@josephmerrill2686]

Rust devil!

[@benedyktjaworski9877]

I don’t like the “reverse dst and src” logic in responses - because you’re given your node id in the init, my assumption would be that you might see a message intended for somebody else and should have some logic for ignoring it (so you should check if the dest field is actually your own id before replying to it). I guess the person who wrote the “you’re still reversing dst and src” comment in the chat at some point had something similar in mind. Since you know your node_id you should use that as the src of the response, instead of just blindly reversing the message’s fields. Of course if maelstorm only delivers messages to the intended destinations and there’s no multicast, it doesn’t really matter.

[@simonfarre4907]

What I'm wonder is, what's there to gain by making everything this generic? I love the generic stuff in this video, as it's a particularly weak point in my skillset, so any practicing on that is great, but every time I am faced with problems I need to solve i just so rarely find myself going for a generic solution - because it's so rare that it actually needs that _level_ of re-use (and in generics case it's not so much re-use, there's some, but not entirely, like in the case of C++ templates for instance).

[@user-nt1kf5cj6l]

missed opportunity to name it Charybdis, for completing charybDistributed system challenges

[@rontman]

Missed opportunity for “chidori”😅

[@act0r399]

Jon, what you think about create new video about Tokio and same field stuff? In many jobs requires Tokio, and still in youtube no cool video about Tokio and async in Rust? Pls))

[@ovidiu_nl]

In the unique ID generation it looks like you did a lot of work to capture your own ID from the "init" message and store in state. But... don't you get your own node ID with every message, including the "generate" message itself? Couldn't you have just used `input.dst`? This is as far as I got in the video, so maybe this will be addressed later on.

[@jonhoo]

Oh, yeah, I suppose you could just grab out the dst field in the message! That only works under the assumption that there are no multicast messages though :p The more relevant response is that we need more of the init state for later challenges, so we might as well construct the flow for grabbing init state properly anyway.

[@ovidiu_nl]

@@jonhoo Fair enough. Great topic, btw! I really enjoyed the video this far.

[@Thisguyrocks518]

I call it seeeered

[@simonfarre4907]

Won't the unique challenge fail? You ran it with only 1 node. Are the different nodes always creating unique ID:s regardless of if they have no knowledge of what the other nodes are spitting back to Maelstrom?

[@ozanmuyes]

Be careful Jon, we don't want you sued... 😒

[@Stopinvadingmyhardware]

No! You will not use Erlang!