Synology's Hybrid RAID (BTRFS+SHR) deep dive - Can I trust it?

[@kernzilla]

Dude, I’m so glad to have stumbled across your channel. You’re format and delivery are ideal 👌

[@sometechguy]

Thank you, appreciate it. 🙏

[@PeterHonig.]

I am glad that I ran across your YT channel, as your videos are topnotch!

[@sometechguy]

Appreciate the comment, thank you!

[@mytech6779]

I would like to know the details of the mechanism they use to do BTRFS data repair with mdadm redundancy. The reason I like BTRFS raid(1) is that I can mount the drives in any other linux machine and use all of the features, so my data integrity has no vendor dependancy. md is also portable of course, but it's that secret sauce glueing the two together which makes me skeptical. (LVM2 can also do a lot of the raid tasks.)

[@Croissant---]

Great work! Keep it up man

[@sometechguy]

Thank you, appreciate it!

[@MR-vj8dn]

Interesting. Not sure about if “md” itself will handle creating a SHR like partition (on a regular Linux machine).

[@manuelthallinger7297]

Isnt this against the GPL from Synology, using btrfs without open sourc

[@sometechguy]

I would think they can fork it, but I don’t know the specific terms for the btrfs code.Many commercial products are built on open source code bases.

[@jensdroessler3575]

Interesting stuff! I‘d like to know details of how Synology migrates from mirror to RAID5 equivalent, and from SHR to SHR2. Do you know any specifics?

[@sometechguy]

SHR will use the underlying MD driver to perform this, so its the standard MD process to move from one RAID type (RAID5) to another (RAID6). The TL:DR on this would be that its a very slow and long process where the parity needs to be recalculated for every bit on every drive in the array from the data of all the other disks. I am performing a SHR-1 to SHR-2 migration currently on a NAS, going from 5 18Tb drives to 6 and this looks to take around ~30 days to complete. And this isn't Synology specific, it would presumably take a similar time on any Linux box with MD running. And looking at utilization during this process, the NAS is not CPU or memory bound, it just looks to be IO wait. I also performed a disk replace to upgrade a disk from 6Tb to 20Tb in a different SHR-2 array, and this took about 15 hours. And this operation is a pure copy operation. I will likely summarize both of these into a video once the migration is complete for those who have the same questions and to see how the process looks in advance of people doing it themselves. Hope this helps

[@DaveEtchells]

Great vid! Is there any at all to scrub an ext4 volume? I guess you could write a script that would copy every file and delete the original once the write was complete. That’d at least force all the file data to go through the RAID logic, which should clean it up as long as you didn’t have more simultaneous errors than your parity would support. I have a total of ~27 TB on a 32 TB SHR volume spanning 8 disks including two spares. It’s for long-term storage and isn’t even powered on most of the time. I worry about bit rot, but unfortunately at the time I set it up, btrfs wasn’t at all stable on Synology. I dunno, maybe I’m just being too paranoid, since I’d think that random flipped bits here or there would get sorted out by the parity logic. (OTOH, best practices say you should rewrite any hard drive data more than ~5-10 years old, so many I should just have one of the AIs write me a big old data-refresh script at some point?)

[@sometechguy]

ext4 doesn't offer scrubbing unfortunately, this is primarily a capability of BTRFS and ZFS (and some less known FS's). But yes, the parity logic is only really to recover from disk errors / failures and not data corruption. There may be system tools that would do the job on ext4, but not sure off hand. Not sure how fast data rots on unpowered magnetic media. Its certainly way better than on NAND but maybe 10 years significantly increases the chance that enough bits got flipped that you have a problem. Maybe its time for some shiny 20tb disks running SHR with BTRFS and copy that data over. :-)

[@DaveEtchells]

@@sometechguy Haha - yeah, a pack of new 20TBs and Btrfs would certainly be a bump in reliability :-) This is just purely archival data from a former business (I tested cameras and lenses for 20+ years), so there’s really no justification or budget for putting any money into it. We’ll see, maybe in 5 years I’ll just buy a pair of 36 TB drives for $150 each, make two copies and forget about it 😂 Thanks for the super-quick answer, btw!

[@mytech6779]

​@@DaveEtchells Scrubbing raid is different from scrubbing BTRFS data checksums which can be of use for identifying damage even on a single partition, though at least two btrfs volumes are needed for repair. The problem with an offline HDD is not bit-rot, the issue is the mechanical parts get stuck and control boards fail so they just won't physically spin up after a few years on the shelf. Best to keep them in a machine that gets booted up every 3-6 months. No controls or fine mechanics is part of the reason tapes are still used, along with the cost per TB when working at very large scale, (Cost for a tape drive is prohibitive at small scale.) but even tapes get copied around every 5→10 years to refresh the magnetic strength before it becomes unreadable, often coupled with checksums and duplicates for repair of detected faults.

[@ntfsguy3601]

CoW is also used by Microsoft's ReFS.

[@sometechguy]

This is really interesting, thank you. I will take a deeper look at ReFS. Seems it’s the future replacement for NTFS?

[@harryshuman9637]

@@sometechguy It's gonna take MS so long to roll it out Windows will no longer be relevant.

[@bartowl]

so much video and still it is not clear how SHR is organized on disks, except that it is handled by md driver. also it is also not really clear if you can have ext4 on SHR, as you state that btrfs is mandatory for SHR and yet there are boxes running ext4 on SHR organized disks... I expected a bit more :/

[@sometechguy]

Sorry if it didn't have what you wanted, but I would say that all videos can't be all things to all people. MD uses standard RAID semantics, and I do have a video that covers how the various RAID types organise data, that explains things like RAID1, RAID5 and RAID6 which the SHR implementation use under the hood with MD. I also have another video with higher level on how SHR organises its data to maximise drive usage. For reasons of time, i wanted to keep the video to a specific topic. There is always more that can be added, and some people want more detail, and some less. On the other EXT4 point, you are correct and mandatory was the wrong word. I have tried to make it clear from the video title and description that this firmly about BRTFS and SHR, where BTRFS is the recommendation for many reasons I mention in the video. But it is, as you say possible to implement SHR with Ext4. I just wouldn't personally.

[@joshuaspires9252]

I am annoyed with the meh performance of zfs so far on my unraid. another issue torrent files are normally dynamic and grow as it downloads thus over ran a couple of my HDD in the array.. so i had to manually move files,, thus i was looking in to BTRFS for my unraid now you be saying don't due to torrents :(

[@sometechguy]

I think torrents should be fine. But yes, probably don't use Copy on Write on the volume for the torrents downloads directory as the files will be constantly growing, and that could result in constantly creating new files. But you can put the completed files on a different volume with CoW enabled. This isn't something I have done, but it seems reasonable. On Synology you can do this not enabling data checksum when you create the volume, but cannot be edited later. Does that help?

[@mytech6779]

Old write-copies should only persist if you tell it to take a snapshot, otherwise they should become free space simply by changing the inode link/pointer to the newly written file. Might be something else going on. You can probably setup a subvolume for torrents with the configuration tuned to what you need. Also, slowly writing a file(slow growing) should be different from modifying a file, but I'm not an expert with CoW file systems let alone the file handling code in your torrent application.

[@joshuaspires9252]

@@mytech6779 oh I have learnt so much since then. At that time I was using the array function in unpaid. It basically sandwiches many single drives with a party drive over then. Thus constant io bottlenecks and small 1tb drives filed easy. Since then I have those 16 1.2TB drives in volume together. So they work as one large drive thus now in zfs2.