The other Kademlia

A look at "Sloppy hashing and self-organizing clusters" by Michael J. Freedman and David Mazières (2002). These are the same authors of the Kademlia DHT paper
00:00 Sloppy hashing and self-organizing clusters
01:22 DHTs are the wrong Abstraction
02:31 DHTs have poor locality
04:30 Le Network
06:01 Kademlia Example
08:47 Clustering
13:50 Sloppiness
16:17 Self-organizing clusters
17:04 Cluster Discovery
19:52 Cluster Merging & Splitting
21:41 Small Privacy Concerns
22:12 Implementation Challenges
23:40 Closing Thoughts

[@bobkelso5681]

It is awesome that you do these videos on such a niche topic. They are exactly what I was looking for.

[@JakobJenkov]

I too have realized that the Kademlia topology has some challenges - e.g. with regards to broadcast / multicast, geographical co-location of nodes, only symmetric scalability etc. To solve these challenges I designed a P2P topology which I call the "Polymorph Polyring" topology. A polyring consists of a central ring of peers, where each peer can have a sub-ring attached, and each peer in a sub-ring can again have a sub-ring attached - recursively - indefinitely. This topology enables peers that are close to communicate quickly, while keeping their data within their sub-ring (which could be located within the same country, region, city neighbourhood etc.). It also makes broadcast and multicast a lot easier, as there is only 1 path to each node in the whole network. No double-receivals of messages - like gossipy protocols have. Each ring (central + sub-rings) can be scaled in size independently of any other rings, so you can do assymetric, local scaling. I am also working on how to implement a DHT on top of the Polymorph Polyring topology. I have some core ideas for that now - but it's not yet finished 🙂

[@n0computer]

Well do a video on HyParView & PlumTree at some point, but it’s our current favorite for building topologies that optimize for latency

[@wesfloyd]

Great video, great production quality 👏

[@n0computer]

Thanks!

[@r.a.a.6472]

I am searching to make a p2p media system to fight censorships. So having a protocol which makes it difficult to get where are the nodes is good. I think that putting rastreability in p2p goes against its own premise

[@davidaronchick-expanso]

Does Node D (in this example) have to maintain an entry for all nodes inside its cluster? That is to say, if it's seeing F, but D doesn't have an entry, how does it know if it needs to cascade to the lower level cluster or not?

[@box-hu6is]

Great video. Just one question: Why not simply use plain kademlia and priorize the k-bucket entries by local round drip delay per bucket? In the end this would create k-buckets with the buckets sorted by node ID's similarity, but the individual bucket entrys would be priorized by round drip delay. A new node belonging to a full bucket would always end up replacing an entry in that bucket if it has a lower latency.

[@louisthibault555]

The entries in your k buckets are not guaranteed to be nearby. The Coral DHT (being discussed here) is good at making sure physically-nearby machines are over-represented in the routing table.

[@IPFSbot]

💡

[@n0computer]

It’s exciting! Complicated, but exciting ☺️

[@marcomunizaga3022]

Great explanation!

[@n0computer]

Thanks!

[@LukeMahan-xr4xx]

What sw is used to show network topology?

[@n0computer]

It’s a vector drawing done by hand (using adobe illustrator in this video)

[@LukeMahan-xr4xx]

@@n0computer thx

[@moonlessiguana1826]

Hi! Is there a way to get in contact with you somehow?

[@n0computer]

iroh.computer/discord