Designing and Tuning for All-Flash Ceph RBD Storage | Ceph Days NYC 2024

Tyler Stachecki speaks at Ceph days NYC about openStack cloud providers and operators. Ceph RBD integrates well with Cinder services and provides very reliable, highly performant block storage. Unfortunately, while Ceph RBD clusters are easy to get up and going, extracting the maximum possible performance from all-flash Ceph RBD clusters is a bit of a black art.
A cursory online search might suggest that one use specific RocksDB settings, place multiple OSDs on each NVMe, or even set some unusual sysctls. However, when tested, we found that most of these suggestions found online are either ineffective, actually degrade performance, or are no longer applicable to recent versions of Ceph. On the other hand, much of the tuning we have found to be effective was only discovered by the community at large more recently, or has yet to be fully realized.
In this talk, we will go over some tips and tricks that have worked well for us in designing and configuring performant RBD clusters using modern versions of Ceph, as well as future possible code changes we believe will extract even more performance out of flash-backed OSDs.
Website: ceph.io/en/com...

[@seccentral]

This numa optimization (at the CHIPLET level) reminds me of coders going above and beyond trying to do the compiler's job. Which is very cool for live streams and tech blogs and tech videos on youtube but it's a pain for actual programmers. So, just like the compiler is supposed to handle the lower level optimizations, so should the software handle the modern architectures properly and spare the sysadmin / storage admin from doing this manual bolt tighening. He's the driver, not the mechanic. I mean it's super cool, don't get me wrong, but it's the coding team's concern (whoever that is)

[@amosgiture]

Complex Performance Tuning can scare a systems admin into opting for a closed source solution but even those end up having crazy bugs that are only known by insiders

[@seccentral]

@@amosgiture not really scared scared, but more like weighing the pros/cons and how much he's being paid cause that sort of decision comes down to tco and negotiated discounts / support packages at the management level. If the company relies 100% on it's own it department and does not buy support and it also has very competent sysadmins, the only reason those closed source solutions are interesting most of the time is just because they say so in a very fancy way and catch the executive's (non technical) eye and there's no code to prove the contrary. So unless we're talking dedicated solutions that include hardware and sla-s, open source is just as if not even more solid than commercial. Back on point, this low level tuning is a pain until you get it and afterwards you just automate away. But still, it is at the fundamental level *not* the sysadmin's job. Just like optimizing branch prediction is not a ts frontend dev's job, tuning for inter-chiplet numa is not the sysadmin's.