Cluster Jenga with

You've heard about Jenga...but what about Cluster Jenga?!
During this year's Creators' Summit, we decided to add a fun twist on the classic game of Jenga.
For those unfamiliar, Jenga involves players removing wooden blocks from a stacked tower and placing them on top, trying to keep it from collapsing.
We decided to use the same concept, but with a Ceph storage cluster! After building redundancy into the cluster, we started pulling components (drives, network cables, power switches) one-by-one to see how long it could stay online (and continue its data transfer) before failing.
You may be surprised at how much this storage cluster endured before it finally gave out!
www.45drives.com
45homelab.com
*WE APOLOGIZE FOR USING POOR RAW AUDIO*

[@lifefromscratch2818]

Coming from the physical security world, this is exactly the kind of "let's try it and see what happens" kind of testing I love seeing.

[@45Drives]

Thank you - it was a lot of fun!

[@LAWRENCESYSTEMS]

Such a great and fun active learning exercise

[@45Drives]

We'll have to bring it back next year but even better!

[@CraftComputing]

@@45Drives Next year, we'll do it on a production server.

[@45Drives]

@@CraftComputing Now we're talking!

[@Demodude123]

Very cool. I am curious about the technical details. If PGs were going inactive and staying inactive. Not sure if ceph HEALTH_ERRs on inactive PGs instantly. A breakdown of "ceph status" along the way would be interesting to get a feel for the topology as Jenga went on. Also how full were the OSDs? It looked like you had a nice number of PGs so every OSD likely had a couple PGs assuming there were enough objects in the cluster.

[@Demodude123]

I also wonder if reads are impacted more by cluster jenga than writes, since writes might be able to go with remapped PGs whereas reads wouldn't have min_size replicas to return a result.

[@CraftComputing]

Ah man, you didn't show my pushups!!!

[@TechnoTim]

Of course I know how SFPs work Jeff!!! 😂

[@CraftComputing]

Just watchin' out for you buddy 😀

[@TechnoTim]

@@CraftComputing Thanks! Haha! I loved the jab, it gave me a good laugh when watching this again (as it did in person too)!

[@RoloSolis81]

This was more entertaining than I thought it would be! Lol!

[@45Drives]

It was super fun!

[@dragonheadthing]

Quite a fun video!

[@45Drives]

Thank you!

[@MarcMcMillin]

this is seriously impressive!

[@45Drives]

Thank you - it was really fun!

[@DS-ou7xm]

👏👏👏

[@45Drives]

Thank you - was a lot of fun!

[@rodfer5406]

Congrats!

[@45Drives]

Thank you!