How to Design a Network Messaging Protocol!

[@blaque5900]

Where can I donate you some money?

[@hoff._world]

cash me outside how bout that

[@tmanley1985]

I just wanted to sleep but this popped up in my feed and now I have to watch it

[@hoff._world]

hey I ended filming at 1:27am AEST so you need to watch it at 1:27am whatever ur time is

[@flame.450]

What resources did you use to learn about this topic?

[@moussaadem7933]

There's no one single resource, learn C, the APIs Linux exposes, a little bit of networking and some programming experience

[@uzumakiuchiha7678]

@whatisaname7962 same bro same

[@flame.450]

@@moussaadem7933 I said resources which is plural not singular

[@RavitShrivastav]

@@flame.450 their point is there is no single resource for each of those. You can’t learn C or networking just by reading a book on each topic. There are some famous books on both of those that you can find with 15 seconds of Googling. Which is the more important skill to learn to build projects like these - the ability to find information online.

[@hoff._world]

ill do a vid on it boys dont worry

[@collatedcopies]

What an underrated channel. Great video!

[@mbakem]

I actually have no idea if we still talking about networking or our relationships but it is right on point 7:08

[@techwithrod]

This is great man. Do you mind sharing any resources/books you recommend for getting up to speed on your current knowledge?

[@scrung]

having a project that incorporates these concepts is usually the fastest way there

[@hoff._world]

vid will come, many people have asked :)

[@Palatonista]

This channel is gold!

[@hoff._world]

so is ur profile picture bro i like it

[@bob-zb3ed]

Okay this is amazing, please dont stop, you are gonna blow up

[@hoff._world]

only cause of legends like you

[@huhulili9021]

Jeez you keep delivering bangers

[@ayushbatra2471]

Subscribed 4~5 days back, already my fav channel

[@yuvraj6770]

loved the thumbnail

[@hoff._world]

I'm one of the artists of our time, arent I?

[@jean-michelcid9594]

I like your videos dude! keep up the good work.

[@hoff._world]

hey thanks! Appreciate the support

[@JW-ns8sl]

Great video, thank you

[@hoff._world]

cheers boss

[@mbakem]

This John and Jenny bit was hilarious 😂 4:07

[@inn6300]

Thank you for this beautiful reference forever and ever

[@iamcasted]

I already love you

[@hoff._world]

xoxo

[@ghostman8163]

Keep going bro. Your videos are very enjoyable and helpful.

[@hoff._world]

appreciate it ghost man. Stay spooky

[@eoussama]

Very insightful.

[@Jay-ke4us]

Hey brotho, I wann reach your level of C programing. I know its needs practice and I should just start doing codes instead of consuming c programing knowledge after knowledge. Is there any youtube channel that do this.

[@blakebaird119]

Yes nice style to this video hoff!

[@mblenczewski]

Nice video. I'd disagree that end-of-sequence markers are simpler to understand and use than length-prefixes (at least for an application-level protocol which is likely to want to be built around arbitrary length byte streams) because you lose packet length information (e.g. for allocating a receive buffer). This is a classic way to introduce security bugs (buffer overflows) and inefficiency. Managing a dynamically sized buffer is always going to be slower than simply allocating the right size ahead-of-time, even if the cost can be amortized across multiple recvs by reusing the larger buffer (although this amortization is applicable to the length prefix scheme too, and requires a single potential realloc() to the marker-delimeted scheme's unknown number of realloc()-s). There is also the need to re-encode the byte stream to avoid the end-of-stream marker in your user-supplied data, as I believe you make a passing mention of. Afaik the space savings in avoiding the extra few bytes for a length field in your header are completely irrelevant for the vast majority of common use cases (maybe in the deeply embedded world with 8-bit or 16-bit MCUs this is still a significant saving).

[@hoff._world]

Thanks for the comment. I work a lot in embedded where we generally try to avoid dynamic memory allocations. I should have mentioned in the video that you can just define a max packet size, allocate a buffer larger and sort of do a wrap-around strategy after you finish handling the first packet, or just reject the message. Even when I'm writing network stuff in the UNIX world I generally will have a fixed sized buffer (avoid reallocs) and have the server ignore a packet that's over a certain size if it's still going after exceeding a limit. Like I said everything is situation dependent. I gave an example of where you are just transmitting text in the ASCII range, which is a protocol I have seen used in industry, where you don't need to re-encode anything. You raise some good points, thanks for the thoughts.

[@mblenczewski]

Fair point, and I completely agree that a protocol should be designed to the specific requirements of its application. In this case, I believe the ASCII encoding forbids codepoints outside the 0-127 range (you have to move to latin-1 or other encodings, maybe utf8 to be future-proof), so you will never see your end marker in a valid user-supplied sequence. I appreciate that in embedded contexts you would tend to define a fixed-size circular buffer into which you read messages, but still think the length-prefixed scheme is more appropriate here because you don't defer the decision to process a packet until you fill your buffer (instead doing this once you read the fixed-size header). Likewise in the UNIX world, you could probably define a fixed size circular buffer out of which to do a form of bump allocation. This is my preferred style as well. Although I dislike dropping packets as I am sure you do, I accept it is a necessity to avoid attackers breaking your server. I will say that the other points in your video are pretty good, and that I look forward to the video of you implementing your home-assistant application :)

[@hoff._world]

No matter which method you choose for length the interesting question arises of how you treat the next data you receive. If it exceeds the length, do you immediately treat the next byte as the start of a new packet? Or do you read until you receive 0x00 in your buffer to let the sender start fresh? All of these questions and the different opinions on them like yours and mine are what makes this topic so interesting to talk about and discuss. For example, in the embedded world it may not be worthwhile to decide to process the packet or not at the start because you will need to fill the buffer anyway over serial if the sender does not stop. There are methods of fixing this issue with both ways, and it is cool seeing what different people come up with :) You may not like my home assistant protocol, it is termination byte to end the packet with multiple payloads separated by a token character :P outside of the ASCII range because that is all I'm sending.

[@mblenczewski]

I see your point. My initial choice would be to assume that clients are all well-behaved and only send valid messages (with the message body not exceeding the given message length), but perhaps in a less controlled environment this assumption does not hold (due to buggy clients that I would say should be ignored). But, as you point out, this is just my opinion as to what should happen, and limits the design space (albeit in ways I would argue are strictly beneficial to both clients and servers :P). Buggy clients and buggy servers shouldn't be tolerated imo, it means peers must be smarter to tolerate the potentially inconsistent messages. Regarding your serial point, I am not sure I quite understand. My understanding is that with pretty much all transports, you can choose to read data coming in from the line and either save it (stuffing it in a buffer), or ignore it (by simply waiting for the next bit/byte/word that comes in). If the transport enforces some kind of framing that exceeds your buffer size (or underflows it), you should have the power to drop the invalid data and return an error to your caller. My understanding is that any restriction to this is entirely enforced by the IO library you choose to use (for example the Arduino stdlib). Perhaps if you work with DMAs or other streams this is different? Although then I would assume you are given constraints on the required buffer sizes by your microcontroller datasheet at least... Please do correct me if I am wrong and/or misunderstood your point. And with regard to your chosen protocol, as long as it satisfies your application's constraints it is fine, and will be interesting to see how I would do things differently at least :)

[@hoff._world]

​ @mblenczewski That point about serial is correct if you're reading one byte at a time, although from my understanding in UNIX it is generally more efficient to read in chunks as you make overall less system calls/context switches, thus with that method you would read into a buffer. On embedded you generally do read one byte at a time over SPI or USART, and you can choose to ignore it as you say, but you are still reading it anyway which was more my point. Your processing would be done after the packet was fully received, so overall for efficiency it is negligible if you store it in a buffer or just 'drop' it and read the next one. It may actually be worse for performance as you would need something like ' if (invalidPacket) { return; } else { buffer[i] = char; }' in your byte receive function and that if statement being run on every receive would be worse than just putting it in the buffer and checking one if condition afterwards. Although I should note perhaps that you would need checks elsewhere to make sure the buffer does not overrun, this might be done in the calling function instead in a for (int i = 0; i < packetLen; i++) or something like that. Thanks very much for the discussion, it is clear you know a lot about what you are talking about and it is awesome getting to hear other perspectives :)

[@norliegh]

WE MAKING OUT THE HOOD W THIS ONE 🗣️

[@stricarzi]

why are bro's videos so 🔥

[@hoff._world]

because people like you are so hot

[@ChrispyChris3]

Pretty sweet video bro, thanks for this.

[@hoff._world]

🫡 you got it chief

[@kevinchen8325]

I clicked on this thinking “I didn’t know tommyinnit did protocol design” lol Great video though!!

[@hoff._world]

hey what can i say im multitalented

[@varshneydevansh]

Bro is cooking

[@hoff._world]

🧑‍🍳🔥🔥🥩🥩

[@xaviervanzyl9299]

johnd is going through his Joker arc rn

[@hoff._world]

@@xaviervanzyl9299yeah johnd didnt implement his async comms correctly and sent jenny a message meant for jessica 💀💀

[@virt_to_phys]

@@hoff._world 💀

[@0lange]

Any networking book or online resource to learn networking fundamentals to do stuff like this eventually

[@hoff._world]

check out my vid "Getting Good at Programming" for a general answer to this question. More specifically you may want to research the IP protocol, TCP and UDP, stream sockets and datagram sockets, and Linux socket programming.

[@simplyexplained875]

Link for spec? Also what software did you use to make it? It looks nice.

[@hoff._world]

just needs some touch ups before I release to so many people, didnt think my channel would get so big will release when I have time :)

[@simplyexplained875]

@@hoff._world take ur time but also dont be shy (i love specs 🫣)

[@hoff._world]

@@simplyexplained875hoff.industries/files/mmp-v2.pdf just for u and @cslearn3044 go easy on me :P

[@simplyexplained875]

@@hoff._world i came thanks :33

[@hoff._world]

@simplyexplained875 xx now make ur own and show me

[@randomsearches369]

I admire your approach and would love to do things as you do. Could you(Anyone) please recommend any books or resources? Thanks so much!

[@royalpotato4733]

sameee...any more resources

[@hoff._world]

have kind of ignored these comments cause I was thinking about how to say in a yt comment.... I think this topic needs its own video so it will come

[@raihanurrahmanjewel9873]

This is the real system design.

[@hoff._world]

I dont build houses but I love being an architect ✏️🗺️👷

[@DydxVT]

let's go new vid !

[@sirk3v]

so happy this is a 2024 video

[@hoff._world]

I can't believe it's 2011 + 13 already...

[@networkedsapien1229]

Noice

[@Awwe12675]

Zooming

[@Nalcot]

This is interesting as fuck my guy. I've always wanted to create a media server that I could do anything I want with straight from my my chair! Content is liquid gold, my friend.

[@hoff._world]

and u've got the midas touch

[@Nalcot]

@@hoff._world me finding this channel is how I know my taste is immaculate

[@AliMoeeny]

and hoff came from the mountain after 40 days and 40 nights, and said "this is toe protocol"

[@collagen1738]

thanks for these videos, i'm the president of my university's linux club and taking a systems programming class right now. i just wanted to ask: how did you learn C and what should I do to get goated like you

[@hoff._world]

will do a vid on it bruz keep an eye out next few weeks top g for being president of the linux club thats pretty cool

[@shaqualeOnale9597]

great video, would love to see an implementation of it

[@lebahsaham9947]

this guy is genius...

[@hoff._world]

that means a lot coming from you, barry bee benson.

[@pistrie8198]

I have no intention to create my own protocol but this video was just too good

[@Graveness4920]

Although its a toy protocol but as a best practice its good to always have first field as version field.

[@BryanChance]

Good idea..

[@royalpotato4733]

Woahhhh.....my brain cells expanded by 2 .....

[@hoff._world]

wow that's a 2x

[@Ozla102]

Your videos are so much fun and chill to watch. I'm hoping to see more, like the actual implementation of the toe protocol.

[@hoff._world]

I just finished all the code, going to record this weekend

[@Ozla102]

@@hoff._world lit

[@MaxTheFireCat]

first

[@hoff._world]

verified and confirmed ✅

[@someonesalt5084]

W video

[@hoff._world]

we're not a11owed to use L's on this channe1

[@someonesalt5084]

@@hoff._worldfrrr

[@aniketnegs]

you have an interesting style. do you have a website?

[@hoff._world]

I do actually, but the massive growth of the channel spooked me and I took it down :P It will be back when it's a bit nicer

[@aniketnegs]

@@hoff._world i get you

[@cslearn3044]

Will you make a protocol soon? I would like to learn to make one

[@hoff._world]

the implementation will come my friend, lemme cook. I have written my own protocol (my home automation engine demo at the start uses it) but need to touch up the doc a little before I release to so many people :P

[@cslearn3044]

@@hoff._world woooot, didnt know there are docs for that lol, can you send a link on where are you looking at?

[@hoff._world]

@@cslearn3044oh I made the doc lmao it's my own custom protocol wait just for u ill release it now gimme 5

[@hoff._world]

@@cslearn3044hoff.industries/files/mmp-v2.pdf here u go bro

[@cslearn3044]

@@hoff._world thanks bruvva, do you have a discord channel or sum?

[@asdasdasdasdasdasd3485]

Keep up the great work!!! I wanted to ask in which kind of software development is this used in? In which programming language would these protocols be written, I am assuming C, and what is the use of writing these protocols for example in a job market, is this useful to know? I am really interested in Linux Kernel Development and that is how I stumbled across your channel and saw also this video and it peaked my interest. Great videos and keep it up!!!

[@hoff._world]

The thing is you can write implementations for them in any language. The custom one I designed I wrote libraries for C, Go, Python and Java, so applications written in any of those langs can talk to each other. Tbh these days you'd mostly be writing custom protocols in something like embedded, if you're gonna use web tech usually you'd just use smth like HTTP requests but it's cool so I did a vid on it

[@sirk3v]

thanks for providing

[@69thApostleOfShindoL]

You are soo cool

[@rogo7330]

This video can generally be applied to file formats too. It's actually interesting to look at old file formats, like MP3, which implement stuff like padding, start codes, etc. This today is just wasting your memory, storage an bandwidth (memory access with today hardware is more bottleneck than just doing some math on CPU), and they just more complex to work with. Something like QOI just does image compression, only abusing the fact that today we count byte as 8 bits and hardware everything optimized to work with that. And it also has magick number in the begining of the file, version and size, which are basically a header. Everything else is just pure stream of data, no packets, no nothing, because everything else is already doing exactly that for you, or you can make it do stuff just wrapping them one into each other.

[@hoff._world]

Super interesting comment. Absolutely yeah, learning about how files are structured and their headers is somewhat parallel to network communication protocols because a lot of the time files will just be streamed through a transport and decoded on the other end raw.

[@benhetland576]

@@hoff._world In some file format specifications the records are even called messages. I have seen some that were sometimes sent to a file, but at other times just streamed directly over the network. Echosounding and meteorological data are two examples that spring to mind, but things like video and audio streams would be good candidates as well.

[@gregoryshields4258]

Nice to see some talk about theory, the creative possibilities for optimizing what goes over the wire to be suited for a given task. Next of course comes the actual mechanism. I'm interested to see what you do.

[@hoff._world]

so am I

[@danraine9009]

awesome channel man I subbed thanks for the video

[@hoff._world]

much thanks my guy

[@Michaeeeel]

Hey! Would love to see how to implement this or any new transport layers in c or cpp

[@hoff._world]

coming soon to a cinema near you

[@sursmansurs9717]

Waiting for part 2

[@hoff._world]

let me cook 😩 🧑‍🍳🧑‍🍳

[@ssahillppatell]

Great video. loved it.

[@hoff._world]

luv u xx

[@didyoustealmyfood8729]

How do you get time to experiment with all this. With uni work doesn't this affect your grade?

[@hoff._world]

The short answer is yes. But it's not as bad as you might expect. For my entire uni career I've been working in industry as a software engineer of some description, which affects my grades more than personal projects like this. That being said my grades are still decent, but I'm just focusing on other areas of my skills (like my actual career in industry and projects like this) which seem to be doing more for me in hiring processes than if I aced every class.

[@nedprin3357]

@@hoff._world How did you get employed in the industry before graduating? Do you apply to normal job listings and just mention that you're still in uni?

[@hoff._world]

@nedprin3357 I'm gonna keep it real with you chief my first job in industry I got because my aunt was the HR lady at an engineering firm. After the first one it becomes a lot easier. I have worked for 3 engineering companies and like I said gonna keep it real they were all through networking. Even with referrals though you need to put your resume forward and pass the interview, it's kind of just a foot in the door.

[@nedprin3357]

​@@hoff._world thanks for the response.

[@marcboutilier7044]

I once tried to implement a file sharing program, and honestly, would've loved to know this beforehand. No, I was(and still) a newbie and wrote the whole thing using python and the socket module. Love that module tho ngl. Couldn't do it in the end, 'cause I didn't know how to write an efficient server or how to receive multiple different pieces of data from one address. No parallel receiving of data, but it was pretty quick in terms of local networks.

[@hoff._world]

still a brave and admirable task. Props for taking it on! maybe in the future you can revisit it :)

[@marcboutilier7044]

@@hoff._world It was mostly a peer to peer system at first, also, how do I receive multiple streams at the same time? or is that not possible.

[@hoff._world]

you might be interested, I have a video where I write a socket server with epoll. Epoll basically lets you block on multiple file descriptors and then handle events for each one as they come in.

[@marcboutilier7044]

@@hoff._world So, it auto-handles a que list? (overly simplified) I did try that, but I was more stuck on the parallel receiving of data to a certain degree, like I could only handle 1 or 2. Although, I do appreciate a lot of the topics covered here, except that I usually send the file size together with the request before actually sending the rest of the file. Might not have though. It's been a while.

[@hoff._world]

depends on how you define parallel receiving. At the base level every network packet your computer receives is serial - it handles them very very fast though. Processing the packet is what is going to cost you a whole lot of time. to 'parallelize' it you might have a thread per client model, where each 'connection' from your socket spawns a new thread. You might also have multiple sockets on different ports or something if you want to take it further. But yeah at a base level your network card receives them serially if that makes sense

[@NicolásMendoza-v8e]

Talk is cheap, show me the code.

[@hoff._world]

with things as complex as this you need to plan before you code. Otherwise you are setting yourself up for a frustrating experience :)

[@NicolásMendoza-v8e]

@@hoff._world You're right, just I think it was like clickbait. But I subscribed and I am anxious to see some kind of implementation. Thanks for sharing.

[@hoff._world]

@user-cd5ft4lb9e well to be fair the title said 'design' and not 'implement' :P don't worry the implementation video is cooking

[@Awwe12675]

Can you please explain thread 🧵

[@hoff._world]

Thread is a length of twisted fibers (usually three strands together), made from cotton, silk, or other material, that can be used in sewing, quilting, embroidery, and other handicrafts.

[@Awwe12675]

@@hoff._world are u drunk

[@txic.4818]

@@Awwe12675you didnt specify bro 💀

[@Shxvang]

@@hoff._world 💀😭

[@Awwe12675]

@@txic.4818 thread programming