Tuesday Tech Tip - How Ceph Stores Data

[@GrandmasterPoi]

It would be nice to make a video with basic daily ceph cli routines. Like creation/modification/deletion/monitoring of main ceph components and parts like osds, monitors, clusters, pools, etc.

[@abhishekpadghan]

I have a query if ceph is back end for openstack and we created a pool for a tenant/project then how we can find which OSD mapped for that tenant/project...?

[@TmanaokLine]

Very well explained, nice and simple. Thank you for making these!

[@colinK85]

Brett Kelly is the man!!!

[@hpsfresh]

OK. Thanks. Now it is finally clear. Three more questions : 1. What is default rule in proxmox? 2. Does ceph reads from multiple sources in parallels (one file/object from 3 osds with triple speed)? 3. For example I make it write to 5 replicas and 2 replicas is minimum. Will ceph see what network is overloaded and write 2 replicas now, 3 later? Or will it always write 5 even if queue is big?

[@GameCyborgCh]

Does it just make 3 copies? How does the crush algorithm choose on how many osd to put the data?

[@45Drives]

Hey! If your pools' replication policy is set to a 3 way replication, Ceph will make 3 copies of that data. When you create your replication policy, you will configure a failure domain, which will tell CRUSH where each copy of the data can live. Ceph's underlying storage is built on RADOS which stores data in objects, so when data gets written to the cluster each object will get replicated twice for a total of three copies across the cluster. If your failure domain is at the server level, each copy will live on a different host. Ceph is always trying to remain fully balanced, so the objects will always be written to the least filled OSD's so Ceph can remain evenly balanced. In the event of a drive (OSD) failure, Ceph will immediately begin recreating the lost replicas elsewhere in the cluster in order to heal itself back to full parity. Thanks for the question!

[@GameCyborgCh]

@@45Drives thanks for clarifying. I'm studying computer science so this is really interesting to me. Could you make a video on doing a cluster with Proxmox VE?

[@Darkk6969]

@@GameCyborgCh I use Proxmox VE 6.2 and it's default settings on creation of the OSD and Ceph storage pools seems to work pretty well on my three node cluster and each having eight 8TB SAS drives for OSD. I do like the CEPH's native WebGUI dashboard better than Proxmox's as it gives you more information so it should be installed as read only just to get more information than poking around in CLI (which I don't mind).

[@sina-nejadebrahim]

you work with ceph cli like people work with linux-you're like really good with it. it would be really great if you guys made a tutorial video on ceph - there is not1 good course that covers ceph and is actually good.thank you .just subscribed and love your content

[@dipeshbendugade8424]

Could you please share , how you have created second rule (type - OSD)

[@abhishekpadghan]

Pls share command in description

[@hayzeproductions7093]

is it possible for ftp server to be installed on the main server? Then any files sent to the main server through ftp would automatically be distributed to what ever nodes the main server sends to?

[@ShoaibKhan_iamsrk]

This is great explanation of CURSH rule and controls ceph gives you to distribute copies of your data across hosts, racks or even data centres!! The only challenge have been facing is the performance, even on the latest stable version of Octopush, performance is just 50% on All flash SSDs. Even reducing replication factor to 2 doesn't improves it. Though there is parallel writes across the hosts (OSDs) it still drops by 50% in performance. Would be great if you could share some insights on the internal writes of the multiple copies across OSDs/Hosts and how the performance is being dropped.

[@bu3ny]

Also, If you use a mix of different storage devices in your cluster, you can customize BlueStore OSDs to improve performance. When you create a new BlueStore OSD, the default is to place the data, block database, and write-ahead log all on the same block device. Separating the block database and write-ahead log from the data and placing them onto faster SSD or NVMe devices might improve performance.

[@gamingparlour1732]

.Icould you please make a video about the source code /rgw in ceph, just some brief explanation will do.It will be of great help to new comers developers

[@manw3bttcks]

I'm a home user that wants to make a small NAS like storage but I don't want all the negatives that come with ZFS/freenas and unraid. I'd prefer to use Ceph but all the videos and documents say Ceph is only worthwhile for many many nodes. But I think their complaints are based on super high performance expectations that probably aren't an issue for home. Suppose I have three small node servers, would you make a video about using ceph in the home using something like three nodes? The advantages I see with ceph is you can add HDDs at anytime, but ZFS requires you go through lots of hoops to do that (grow the pool by resilvering over and over or adding a new pool.

[@rinat1985]

When there's only one host serving as a NAS, why not ZFS? Suppose you have 6x4Tb hard disks, then you could create raidz-2 that would serve for a long time and when it comes to upgrade, you can simply put temporary new 6x10Tb in a PC , create new zpool and do 'send/receive' operation, then export new zpool and place/import it in an old NAS box (OS there have to be on a separate drive from beginning) I see this as a pretty simple and straightforward configuration About your example, I don't know much about Ceph, but for clustering Proxmox team highly recommends using even number of hosts (4+) with additional host for adding votes to complete quorum to be able to go if one of the clusters host or this particular 'vote' host goes down, maybe for Ceph it's not the case... but what is the case is that such configuration requires at least 2 separate network and at least one of them (private one) should be 10Gb/s (better to have redandancy for the private network also in case of switch port failure) and you have to buy appropriate switch, UPS (not cheap for only correct shutdown, it should be able to give power for an hour) and so on... It'll soon become some kind of Ceph home test lab rather than NAS4Home...

[@manw3bttcks]

@@rinat1985 Why not ZFS?, you perfectly pointed out one of the big problems of ZFS, you can't just add a single drive. The expansion method you just explained required you to buy 6 equal size drives and then send/receive to those. What a lot of people like me want is to just have a system that can add drives one at a time and of any size. That's one of the advantages of ceph, you can just add one or two or any numbers of drives and they don't have to match in size.

[@mdd1963]

@@manw3bttcks So we envision going to the trouble to have 3 or more hosts at home, but, only wanting to add one drive at time for expansion? That's funny. :)

[@juz882010]

@@rinat1985 10TB drives? that raid rebuilt is gonna suck

[@rinat1985]

@@juz882010 there's dRAID option now, rebuild can be distributed

[@georgelza]

... curious, any chance there is virtual box vm's available to run on a strong laptop to learn CEPH ? aka for say something like the above...

[@NTmatter]

If you can't find a pre-made image, you can set up one or more of your own Linux VMs, and then follow through one of Ceph's installation guides (1). Remember to take a snapshot once you've installed the base OS, and you can roll back to the clean state. This allows you to reinstall Ceph if something goes wrong, or if you want to try again. Alternatively, you can compile the source code (preferably in a container) and use the vstart.sh script to launch a local dev-mode cluster. It takes a while to compile, but it's interesting to see how things get built. 1: Installation with Ceph-Deploy docs.ceph.com/docs/master/start/

[@georgelza]

@@NTmatter I know I can build the VM's myself, allot of things can be done self by following and hoping you get it right, or ... the idea of a downloaded VM is a great way for something that already done this, perfectly, and works make available .

[@juanc4065]

Thanks man

[@samirariazi9580]

Thanks

[@juz882010]

dont you waste alot of IO replicating everything? even multiple times? that extra IO on flash, does that not make them die faster?

[@Evan-nq3de]

Oh no Brett has the Corona!

[@AxelWerner]

You clearly never did a screen presentation before. F for failed.