Make Your Home Server Go FAST with SSD Caching

[@trapexit]

mergerfs author here. Thanks for the coverage.

[@JonathanYankovich]

Mergerfs user here. On behalf of so many, THANK YOU!

[@JonathanYankovich]

(Also, NFS shares on Unraid using mergerfs are broken/unstable. It might be a bug in libfuse, not mergerfs, but it’s causing me to ditch Unraid and roll my own snapraid+mergerfs probably on Ubuntu)

[@benjaminshtark5977]

@@JonathanYankovich never so many, owed so much, to so few...

[@myghi63]

Thank you! because of your videos I learned a lot about server stuff and also improved my own server here! I already have a nas with a corsair nvme drive and probably in 2023 I will be able to switch to 2.5Gb/s network. Btrfs has been my FS of choice on all my OSes at home and on my server it's running at RAID 1 + zstd:3 compression, without any problem at all on two seagate ironwolf 4TB drives

[@muazahmed4106]

When will you do a video about home automation?

[@rashedobaid]

Up, Looking forward to this

[@tomlloyd7122]

Also want this

[@TheTeregor]

Small note about BTRFS: its RAID1 is not actually RAID1, it's a different type of RAID that is confusingly named "1". To cut to the chase: BTRFS RAID1 (and RAID10, for that matter) can tolerate only ONE disk loss, REGARDLESS of the amount of disks in RAID. Please consider this before committing to BTRFS on your NAS. Suppose you have 2 6TB drives and 1 8TB drive in a BTRFS RAID1 (yes, you can do odd number of disks, and different sizes as well). Now you write a 1TB file to it, for the sake of example. The way BTRFS works, it will write 1TB file to the "most free" drive, which is the 8TB drive. Then, it will write a copy of it to ANOTHER "most free" drive, which is either of the 6TB drives. Let's write 1TB files until our BTRFS RAID1 is full and the free space on disks: 6TB#1 | 6TB#2 | 8TB 6TB | 6TB | 8TB 5TB | 6TB | 7TB 5TB | 5TB | 6TB 4TB | 5TB | 5TB 4TB | 4TB | 4TB 3TB | 3TB | 4TB 2TB | 3TB | 3TB 2TB | 2TB | 2TB 1TB | 1TB | 2TB 0TB | 1TB | 1TB 0TB | 0TB | 0TB We can see that our biggest disk (8TB) is used the most, while its free space doesn't become equal to another two drives' free disk space, and by then writes are balanced equally between the disks.

[@Felix-ve9hs]

15:11 The ZIL (ZFS Intent Log) is a part of the ZFS Copy-on-Write function (for preventing Data loss), which only gets used with sync writes (e.g. if you use your Storage Server for Virtual Machine Storage). On normal file copy operations, the ZILL never gets used. If one does have a lot of sync writes, they should put their ZIL on a dedicated Log Device (SLOG), which would usually be an SSD. The L2ARC is an extention of the ARC read cache, wich will cache frequently access files in you free/unused RAM. a L2ARC will be useful if you cannot fit all of your data in your RAM you want to be cached, but it will only speed up operations that access files already on your ZFS pool.

[@agistan7764]

Great and 100% correct explanation. I'd also add that ZFS is great for handling lots of small files and has outstanding data reliably and safety. However most of it's features really be useful in enterprise (like snapshots and replication)

[@chris11d7]

While you're right about needing a network upgrade to get more sequential performance from SSD caching, but even on single gigabit, you're getting a huge performance advantage in random IOPS. I love your videos, keep them coming!

[@lawrenceheine3458]

😢

[@chris11d7]

@@lawrenceheine3458 what?

[@robertt9342]

I agree, I would be using it for iops and low latency.

[@nichtgestalt]

Thank you very much for this and all the other videos. Even though I don't use a server (yet?), it is so interesting to see these tutorials, especially the ones about power efficiency. Have a good one!

[@Guilherme-qk9so]

Your videos are always so helpful and well made. Thanks for sharing this!

[@halbouma6720]

There's a linux kernel driver that does tiered storage as well - its called btier. It moves the more often used data to the ssd drives. I use it, works great. Thanks for the video!

[@abwesend182]

can you give more information about this topic? maybe where I can read more about it?

[@kevinwuestenbergs7612]

Not really mentioned in the video but cache in unraid is read/write only when the files are in the cache pool. After the files have been moved back to regular storage you have to manually move it back to the cache. I would think tiered storage should do read and write caching but in unraid it only does write caching.

[@bertnijhof5413]

12:10 The memory requirements of ZFS depends completely on your use case and the mentioned rule is valid for a server with many users (say >20). I use ZFS on my desktop and laptop (Ubuntu) and I limit the L1ARC cache to 20% to 25% of my RAM size, mainly to save some time starting VMs. If needed ZFS will free up cache memory, if programs or VMs need it. On my 16GB desktop I limit the cache to 3 or 4GB; on my 8GB laptop I limit the cache to 1.5 to 2 GB. On my 2003 backup-server (Pentium 4; 1C2T; 3.0GHz) with 1GB DDR (400Mhz) I did not set limits, but FreeBSD/OpenZFS limits the cache to 100/200MB. Currently I use a 512GB SP nvme-SSD (3400/2300MB/s) and a 2TB HDD supported by a 128GB sata-SSD cache (L2ARC and ZIL). Often I run the datasets on the nvme-SSD only caching the meta-data (L1ARC), because full caching only speeds up disk IO by 10% to 20%. That small difference is, because of the fast nvme-SSD and my slow Ryzen 3 2200G, who needs relative much time for compressing/decompressing the records. I don't complain, because I boot e.g Xubuntu 22.04 LTS in ~6 seconds mainly from the cache or in ~7 seconds directly from the nvme-SSD. Note that all my data storage and all transfers of the changed records during an incremental backup are lz4 compressed. The ZFS snapshots on my desktop saved me twice from 2 hacks I've experienced this year.

[@anthonvanderneut]

I have been using Btrfs over mdadm raid-6 on several servers, for many years now. Apart from the occasional dead drive that needed replacing this worked without problem. I did switch to combining smaller (1Tb) disc partitions of each of the 6 drives in a raid, then combining the raid with LVM and put Btrfs on top of that. The smaller disc partitions have the advantage that I can run a raid check over night on them, instead of having a huge one starting Sunday morning early, but not finish until very late in the day, making access very slow. Of course some people will roll their eyes about this layering and the loss of speed, but for me and my usage pattern that is not a matter of concern.

[@ciaduck]

A better way to add disks to ZFS is do add another vdev (array) to a pool. Yes, you should not simply add just one disk, but by adding pairs of disks, or even another raidz volume with several, you can grow your storage pool without having to do the kinds of gymnastics you outlined. ZFS will stripe across vdevs in a pool. It is a trade off in convenience and cost (requiring multiple disks), but what you get is security. ZFS, essentially, is designed to force a user to do it the "safe" way. This will also let you avoid the MergerFS on ZFS thing you alluded to. Which sounds like a bad idea.

[@shawn576]

Stablebit Drivepool now has an SSD caching option. Things saved to the drivepool are saved to SSD first then moved onto HDD after.

[@NN-uc1fh]

Thanks for your tips and videos. I like you're channel for helping me in my daily life here as an non-programmer. Greetings from southern Germany !

[@lizzyfrizzy4969]

Im really glad you talked about NAS. I want to build a medium sized (50tb)array, but out of NVME-m2 drives. For pure speed at all costs. In my research i found that due to chipsets, the fastest raid arrays for m2 are quantity of 3 drives. Although some boards and many m2 raid cards have 4 ports, actually adding the fourth drive lowers the performance. Therefore, each submachine of the array would be limited to 3 drives. Although i could plug in 4 cards with 4 drives each, those 16 ssd wont enjoy the full bus rate (i think?) Therefore, instead of building one server with 16-24 drives, the m2 ultra speed nas would need to be made out of a network of machines. Ideally built with the smallest board with a fully supporting chipset i can find. Well, i know nothing about networking. What manages this nas array? Another machine with tb of RAM? Building a high speed high performance array is more complicated than i thought😢

[@ShaferHart]

living it up before end times aren't we

[@chromerims]

This is timely for me. Thank you. Will watch tonight.

[@rasaskitchen]

I learned a lot about RAID in this video than 5 years running a server.

[@LXJ1974]

Excellent video as always. Thanks for that

[@bradlee5374]

Wolfgang, do you think you could make a video about your current OS selection for your server? When you were upgrading to your server rack you said that now that you have CMR drives you will try trunas but I see you are using unraid now. I think it would be great to hear your thoughts about unraid vs trunas core/scale vs Ubuntu server and see the process of how to switch your OS on a current server.

[@kdog8787]

You can utilize CAT 5e with 10g; it's just not guaranteed to work. I've done it. Ironically, I had trouble with CAT 5e (same cable) with 2.5g because I was using low power NICs.

[@tredonlinder2543]

Thank you for your work! Keep rolling.

[@hippoage]

9:50 Looks like one of options is missed: LVM. It also can organize RAIDs and cache on SSD.

[@ZeroXtiK]

1min of video and I love the idea, love ur vids dude!

[@jamesbutler5570]

Running btrf in raid 6. Upgraded harddrives with bigger ones, switched defective drives, changed fron rait 1 to 6. Now its bigger 50TB. Never had problems for more that 5 years

[@ИгорьБотухов]

Thank you veeeeeery much for the video about homelab💖

[@adriancozma6102]

Very insightful, thanks for sharing!

[@shanent5793]

zfs works with diverse drive capacities and you can add any number of drives of any size to a pool, without having to destroy the pool. It's been that way for 10+ years. If there is any limitation, it's in the management layer, and not in the filesystem itself

[@ShaferHart]

for all intents and purposes it does not support it. Sorry.

[@baricdondarion6228]

This is my best KZbin channel. Straight to the point, no BS. No irrelant talk. Hit hard on just the important information. I found out despite always getting your videos in my feed, I wasn't subscribed. Totally fixed that.

[@wagmi_dude]

I strongly recommend server grade HBAs in IT mode rather than cheap chinese SATA cards. I own H310. It has 2 6Gbit 4-port channels. For SSDs there are similar cards with 12Gbit throughput. Originally I used to run my ZFS on 4 port SATA controller but it frequently made disks resilver.

[@Akshun82]

Those Orico adapters are awesome.

[@Nalanaij]

This is gold! Thank you! My relevant data is on my Desktop machine and is synced to the nas and Notebook. So i'm Independent of the cable Speed, but my data ist in the nas. Though, i'm no Video Editor and this isnt the best idea for large quantities of data. Hope tiered caching support comes to truenas. I'll check the level1forums

[@mrsansiverius2083]

Guys he's not changing haircuts, a new Wolfgang periodically appears and kills the old one, taking his place.

[@robertt9342]

It’s pretty standard stuff.

[@Kohega]

Very useful documentation, thanks

[@benedikt3880]

what made you switch/extend your homeserver to UnRaid? I remeber you said in your old home-server tour video that you consider UnRaid too limited. Maybe that would be an interesting topic for a video as well.

[@ShaferHart]

I'm interested in hearing his rational for the change as well. Maybe he did and the algo hasn't pushed it to me lol

[@himmelsrand7527]

If someone wants to delve deeper into networking, I wouldn't recommend an unmanaged switch (can't use VLANs etc.). If you're unsure i would still go for a managed Switch, you save yourself from buying another Switch in the future (if you eventually need Management) and the price difference to an unmanaged switch is negligible. A Switch i can recommend for smaller networks is the Zyxel XGS1210. It has 2 x SFP+ Ports, 2 x 2.5 Gbit, 8 x 1 Gbit RJ45 Ports and is managed. You sometimes can get it for around 120€ (Currently 160€ in Germany).

[@WolfgangsChannel]

Great suggestion! I personally bought a 'smart' switch only to discover that pretty much all of my client devices support tagged VLANs. This Zyxel switch is a really good deal though

[@cinemaipswich4636]

I thought hard about adding NVMe drives to RAM overflow (or VDEV Cache/L2ARC), but it worked out cheaper and faster if I just bought another 128GB of ECC RAM. My TrueNAS server uses much less resources, by having fewer devices attached. As for other VDEV's (metadata, log, dedup), I will think about later. Network speed is my next project. I see the that I do not need a switch if I use direct attach via the SFP cable. I am a single user. Thank's Wolfyie.

[@kz2682]

I use mergerfs and snapraid for my 11HDDs, so only the Hdd is running which is in use, this will reduce the powerconsumption.

[@JasonsLabVideos]

good video. 10gtek makes nice compatible dac cables !! Use them for my home lab & at work no issues at all!!

[@tabimeterable]

Hey, nice Video *thumbs up*. An alternative to an mergerfs/copy based cache is bcache. You just have to write a superblock to the drive in front of an ext4 Partiton, or let bcache handle that with a new drive, and then group the drives under mergerfs. Works flawless for me and is stable, well tested and in the kernel since 3.10., drives can be "easily" add or removed to the same cachedrive and you can excess the ext4 partiton by just offsetting the mountpoint? (or something idk, but it works).

[@tabimeterable]

And it is highly recommend to mirror your cache drive (eg. 2 drives with mdadm Raid 1) since a failed (non raid) cachedrive is pretty bad.

[@nekoskylynn]

Just my two cents about 2.5/5 switches and fiber cable confusion: In my country (it is Russia, please don't hate me) 2.5 or 5gbps switches cost like x2 more than 10gpbs switches (8 sfp+ ports!!!). I've been planning to setup 2.5 for a looong time (pcie cards are dirt cheap though). After long managing I just got bamboozled and set up 10 gbps network everywhere lol. It was hard af to manage which DAC cables are compatible, which fiber cables to use, which transceivers to use. Uhhh. Ended up with LC OM3 Multimode fiber cables and basically every transceiver compatible with LC/mm fibre cable. And to answer to all guys to question I couldn't find: Yes, most DAC cables are vendor agnostic and you will not find any trouble. I am currently using TpLink/Zyxel/Cisco switches together and all my random DAC cables are working (Cisco, nonane, aliexpress cables lol). One problem with DAC is that it requires more power so more is power consumption, but its great for rack/datacenter solutions because its short and less prone to damage Thanks for the video! I wish it would be here earlier lol. I am currently using mergerfs/snapraid and looking forward how it goes with tiered caching

[@TuMbl4]

So, how it goes with tiered caching? Did you try it? ;)

[@Nunoflashy]

Why would someone hate you, is it because of the war, something that you don't have any direct participation in? (Unless you're in the army, of course). I get it that this is the internet and you get insulted or canceled for these petty reasons, but having to apologize for such a thing astounds me. If anyone hates you for this, and you have no connection to the war, which you most likely don't, then you owe no one an apology and it's great that you found them out so you can stay away as a result.

[@brookerobertson2951]

I run deathwish RAID. “RAID 0”. I also run suicide Linux as the OS. “It deletes the whole system of you enter one incorrect command”. It’s like driving your car at full speed with no seatbelt. But it’s okay because its only hospitals computer systems not my personal. Makes my boring IT job way more exciting.

[@loadmastergod1961]

working on upgrading my network to 10gig now. not sure how good it'll be with the short cat5e to my porch, but once i heal from surgery and the ground thaws, i'll run a new cat6 line to the garage again and have full 10 gig network to my servers

[@Airbag888]

Love this series...

[@BTA_KeepItFun]

Found your channel by a happy accident. Very helpful and well written videos! Would be interested to hear your take on OpenMediaVault (OMV6) if you've used/checked that out. Personally been quite happy using OMV for some years now, but the new UI is downgrade to previous (OMV5). Happy early Winter time =)

[@secretcactus4717]

So you changed your server's OS to Unraid or did you use forbbiden black magic to use two OS at the same time? PS: Your home server videos are great, keep it up!

[@AaronMolligan]

Thats setup looks nice android the information you can presented is really helpful. I went a different route and currently have my nas setup in a HCI solution with one nvme as well system cache. I prefer to use unraid but currently being forced to use truenas for my companies setup

[@YannMetalhead]

Great video!

[@Zavala-z9g]

Thank you very much.

[@wheisenberg559]

Hyper-V Core does also support tiered storage with Storage Spaces.

[@L0rDLuCk]

ZFS is King! I use it since 2007 and never lost a single file even though i lost multiple hdds in the past 15 years. it is by far the best file system on earth. no one should consider any other filesystem. if you put enough ram in your machine you also dont need any ssds for caching. i saturate a 10gig link with no problem for any file transfer only having enough ram. also the scrubs is much faster with more ram! and even linus accepted that unraid is crap and shouldn't be used in any situation!

[@ShaferHart]

Without the data being mirrored/paritied in a btrfs raid I see little reason to use btrfs for your media (which is probably most of the storage). You're not going to snapshot largely static data and you're using snapraid for parity/backup anyways.

[@sarundayo]

Came for the speed, stayed for the Sanic memes

[@alekzandru221]

Got 10gbe, 3x cards with trans included 24x1gb 4x10gbe switch, om3 cables all for under $300. Found an aruba 1930 for 100, and cheap hp cards, had to do some driver updates, but all worked out.

[@ivosarak959]

If the DAC is desirable, but a bit short then the AOC will do the trick as well and with much longer runs.

[@HueMongus101]

The Supermicro 3.5" bays work with the Dell server 2.5" to 3.5 metal bay adapters. Much cheaper than the Orico

[@Bixmy]

the thing with 10g is if only you need the 10g on the window machine just get 2 10g and direct connect them and manually assign IP not related to the dhcp.and have other connection go through 1 or 2.5 as usual

[@Bixmy]

you could add 10g switch later too if you want it for other mahine

[@Bixmy]

with this method you could only pay like 60 USD for a direct 10g connection from workstation to nas. well this will only work if you using the nas along thou

[@Bixmy]

a 1m dac for 20 USD 2 10g nic for 20 USD each so 60 USD

[@roysigurdkarlsbakk3842]

CAT6a is rated to handle 10Gbps over up to 100m, so it shouldn't be a problem.

[@MarkJay]

When I look at some of my sata ssd specs, they say 5V at 1.5A, or 7.5W. That seems just as high as a 7200rpm HDD

[@keshavrathore4189]

Thanks

[@pieter-yt]

Ive been running 2 2tb nvme drives in raid 0 for a few years now and its great OFC i keep daily backups off the whole system in case off a failure Still dont recommend doing this unless you like living on the edge waiting to be kicked off it and have to recover ur whole system from backup :3

[@ShaferHart]

that's child play bro

[@mikedoth]

I need to get sponsored if I want some sweet hardware :-)

[@MrCoffis]

In which video did you talk about tiered caching? Will you ever consider doing how to's in the future?

[@MasterCommander3]

I like to first say I love your content and your opinion but I would like to provide counter arguments to your opinions since this serves as a guide for people who are getting into servers. I would love to hear yours or others opinions on these points below. First 2.5Gb is not worth the upgrade from standard gigabit. Anyone looking to upgrade their NAS should go with at least 10Gb. At 2.5Gb you are throwing away money at such a minuscule jump in performance. In your price comparisons, 2.5Gb is 50% of the cost of 10Gb at only 1/4 of the performance. Compare that to 40Gb or higher and that value suddenly begins to drop. Secondly, suggesting unmanaged switches should not be recommended for a NAS. You want to have LACP and Vlan support to be able to increase bandwidth and isolate networks. A NAS will greatly benefit with LACP or LAGG to provide link redundancy and increased bandwidth to the NAS. Such as two 10Gb devices can now communicate with your NAS at full speed if the NAS has 2 x 10Gb links. VLans is also a must for a NAS to reduce the risk to unwanted devices to your data. Last thing you want is for your unwanted visitors accessing your wifi or unsecured ethernet jack. Thirdly, while 100Gb is just barely out of reach for consumers, even with Mikrotik's new affordable 100Gb switch, Qsfp should be the "real" option 2 between 10Gb and something else. 40Gb switches are not used in the industry anymore and corporations are selling their inventory for next to nothing what it was worth. 40Gb should be recommended for anyone using NVMe drives as it can effectively use one NVMe 4.0 drive with a single 40Gb link. Since NVMe is priced the same as Sata SSD's, the cost of a (NAS system with 10Gb + Sata) vs a (NAS system with 40Gb + NVMe) is only the cost of the 40Gb switch and NICs. Fourthly, you are correct about NVMe is only a few times faster than sata and if someone were to use it on a 10Gb or less system they wouldnt be able to use all the NVMe bandwidth. However, NVMe drives have much much faster IOPs compared to sata. A sata drive typically has 70-100k IOPs and NVME has 900k+ IOPs. Even if the NVMe drive does not get used to its full bandwidth in sequential R/W the NVMe will outperform in where it counts in IO performance. Lastly, you are entirely wrong with one point you made about ZFS. That is, you can increase your array size. With ZFS there are two things that create an Array: Pools and Vdevs. A vdev is where your drives are and used with a standard RAID configuration: 0,1,5,6 etc. A Pool is where your vdevs live on and each vdev is split evenly with how data is spread out. A Pool can also have mixed sized vdevs exist inside it and have additional vdes created without losing data. So yes you can 10000% increase an Array in ZFS. I would love to hear your opinions on my points and I am not afraid to be wrong. If anyone has any additional comments or discussions to be made I'd also love to hear this. HomeLabbing is an amazing hobby and absolutely love the support this community gives to others to learn and understand.

[@WolfgangsChannel]

Thanks for the input! I agree on pretty much all points. However, when it comes to ZFS, you can’t extend a RAID pool as easy as you can do it on BTRFS or MergerFS. You can create a new vdev, but you’ll need at least two drives for a RAID1 vdev and at least 3 for RAIDZ.

[@uncreativename9936]

VLANs aren't necessary at all if you only connect your server and workstation to the switch and block traffic with your other networks at the router. Although I think even separating your server from the rest of your network is unnecessary, just make a guest wifi network and forget about it.

[@tredonlinder2543]

Please make a video about Unraid (comparison with Ubuntu setup)

[@BazShe]

sounds like a good way to kill the flash drive if you are doing constant writes.

[@neverwasthere]

It sounds like you talked about SSD cache for software RAID setup. But how to implement SSD cache on a hardware raid 5 setup. I have a ThinkServer TS440 with Windows Server 2019 on a separate SSD and Data on 4x 4TB HDD wired to a LSI 9364-8i hardware RAID card with RAID5. How to add SSD as cache and how to enable the cache from where? the hardware RAID card control or OS?

[@brandonedwards7166]

It would probably be better to jump up to 40gb based on price. dual 40gb cards are about $10 on ebay. 40gb 8 port switch for $40. It is a little more to configure but faster and cheaper.

[@WolfgangsChannel]

Yep, but careful, older 10+Gbit cards run hotter and draw a lot of power

[@sufyspeed]

You could also use fibre instead of Cat 6A

[@hiasi94]

First really good channel with great content I am rebuilding my network and want to get a new switch anyway what do you think of 1Gigbit switches with 2 SFP+ ports for 10G?

[@ankitsinghaniya]

Seems like ZFS can now support different size drives in the pool and also add/remove after initial setup?

[@robertt9342]

Unraid now has ZFS support in 6.12.0

[@mamdouh-Tawadros]

Forgive me for a simple question, if you have a SSD to boot from, you can still benefit from another cache SSD?

[@tomdillan]

If your motherboard has dual gigabit nic’s how to do combine (bond/bind) them to one for faster transfers?

[@WolfgangsChannel]

Yes, as long as those two ports actually go through two separate NICs. Your switch will also need to support port bonding/binding

[@joegaffney8006]

Random read and writes can still benefit from ssd caching even on 1gbit

[@smitler]

Love your videos! I'm quite new to diy server/home networking, but have enough experience to follow-along and learn from you, but I have a question: So you're running unraid as you're main os for your server right? And what file system are you using with Merger/SnapRaid? I currently have my hands on a Cisco UCS C240 M3 (2 x Xeon E5-2600 CPUs & 64gb 1866mhz RAM, 4x gb NIC) server with 22x 2.5" sas drives (a mixture of 900 and 1.2tb sizes) along with a PCIe onboard dual sata card (that I'm running 2 x 240gb SSDs). I'm currently running Ubuntu with Merger/SnapRaid and I'm mainly using my server for JellyFin/Home Assistant/Frigate (for my CCTV NVR storage and object detection). I've picked up a lot of tips from your videos but now I'm confused as to whether I should be running Unraid OS rather than struggling with Ubuntu and manually setting up everything there as my Linux knowledge is limited and I'm taking a lot of time getting things to work with each other.. I guess I'm also trying to see if I'm utilising my Setup to the best of its ability and would love your take on it. Where's the best place to chat to you about this?

[@WolfgangsChannel]

Unraid is great if you want a "just works" solution. You can get a trial license key and see if you like it before committing to it

[@smitler]

@@WolfgangsChannel Awesome, thanks buddy! Also what file format system do you recommend to use with MergerFS ?

[@WolfgangsChannel]

I use XFS for hard drives and ext4 for SSDs. You can also use ZFS for SSDs

[@Bofrostmannful]

I bought 2 10GBit Cards and a 3m DAC at eBay Kleinanzeigen for 100€. Maybe the cheapest way top Connect the Main Desktop with the NAS.

[@swistak0220]

I was just thinking about all this. I wanted to go with managed switch but markup for 2.5 GbE is huge. I also want to keep my ZFS pools so I found the autotier by 45drives. Unfortunately there is not much info on this.

[@Felix-ve9hs]

Why does one need 1 GB of RAM for every 1 TB of storage with ZFS in the first place? I have no problems running a 18TB pool with only 8GB of RAM, I bet it would work fine even with only 4GB or even 2GB of RAM... :/

[@hydrophobicwalrus749]

I also get away with running 16 TB of storage with only 8 GB of RAM. Suboptimal, but I haven't scraped the cash together yet to upgrade the RAM. My guess is they're leaving open the possibility that you might use ZFS deduplication? Otherwise, my little server chugs along just fine (although it's only responsible for very light tasks).

[@Felix-ve9hs]

@@hydrophobicwalrus749 I dont plan on ever using dedublication, its just a NAS for archiving and backups, as well for NFS targets and SMB shares. Nothing special but runs rocksolid. :)

[@hydrophobicwalrus749]

@@Felix-ve9hs Lol. Same use case for me as well, plus a little Plex on the side.

[@austinbarrington778]

Yeah that doesn't make much sense, use as much ram as you think you will need. The more you have the more it will cache and the more useful it will be. Just watch what you configure it to use. We use it to decrease disk latency for high workload environments.

[@gold-junge91]

Can you show us your unraid setup? I have seen you have a Time Machine backup on your unraid system, I have play 2 years ago with unraid and only little experience about it

[@torgeirarny6642]

The statement "you want atleast 1gb of ram for every 1tb of storage", is simply wrong. ZFS is not RAM hungry at all. It just do not let any RAM go to waste. It will use whatever it can access. Allthough there is a minimum recommendation on how much RAM you should have regarding a Zpool, but that is not 1gb RAM for every 1tb of storage.

[@kdcadet]

With zfs, i believe you canadd new v.evs to a pool at any time, but you cannot add new drives to a vdev once created. Also, inside a vdev you should have the same drives, but another vdev can be totally different, although i believe not always recommended. It's complicated and i am not sure i understand it myself

[@vipvip-tf9rw]

3 drives is enough for any home usecase

[@derekp6636]

yep you can add vdevs, but removal is an issue so set your stripe from the start. For me I started with 6x zfs raidz2 and it was cheaper to add another vdev of 6 disks vs trying to restripe the existing to larger. Helps if you have a ton of disk slots though.

[@ShaferHart]

@@derekp6636 I'll take my chances with snapraid parity lol. Thanks.

[@benjiderrick4590]

Thanks! My home server is running openmediavault right now, so I don't know if I will be able to set up mergefs on it with a pair of small SSDs. Right now the focus is on power efficiency and savings, especially since my ISP doesn't let me route my ports to the internet, so I have no means right now to use jellyfin, Plex or even ssh

[@ShaferHart]

you can't port forward on your router? how come? You can also look at tunneled solutions like tailscale or cloudfare tunnels.

[@benjiderrick4590]

@@ShaferHart well that was long before I knew I wasn't on a full ipv4 stack, which I asked to my ISP. I now run jellyfin (and all other containers) through tailscale whenever I'm outside, but initially the plan was to do reverse proxy. 8 months in, and I now have doubled the storage capacity and added a "cache drive" to help reduce power consumption (and wear) when watching animes and such. Well for that last one, I wish I had chosen truenas instead, it would have been much more versatile ; here I have to manage copies of media on the SSD drive

[@KristerWarmenius]

is there a way to setup a SSD-cache at a low level, ie. in BIOS so the cache works on sector or block level?

[@pipeliner8969]

you are so smart

[@louisunruh2386]

I'm trying to do ssd caching on linux ubuntu server without unraid. is there any option to implement the unraid behavior? In a what's on my server video a couple of years ago you said you were using ubuntu as os. Did you have tiered caching there aswell?

[@michaelkaercher]

Not sure if I like the approach. I am not really willing to steer the cache/log manually. I find it preferable to have the decisions done by zfs based on proven methods. I went for 2.5g network card. Good enough for 2k video editing.

[@WolfgangsChannel]

It's not 'manual' per se. You can set up your 'mover' script to either dump the data from SSD to the HDD array when it gets full, or the other way around - pull the most newly accessed data from HDDs to SSDs as space allows

[@kazumakazuma5814]

Getting a usb-network-card with unraid can be an issue depending on the chip. 6.10 or so broke the drivers for mine.

[@rockerbacon]

Important thing: the write hole bug is not a BTRFS bug, it is a downside of the parity system. No matter if you’re using BTRFS or ZFS or anything else, if you’re using RAID5, 6 or anything else which uses parity, there are routines you need to put in place in order to protect yourself from parity data loss issues.

[@michaelhenry3234]

Not exactly true. ZFS out of the box isn't vulnerable to the write hole. No additional routines required. That's due to CoW, ZIL, etc.

[@ShaferHart]

it's a btrfs bug when the only fs affected is btrfs. Go fanboy somewhere else.

[@BA65574]

Not true at all. ZFS is architected differently as a CoW filesystem. It is a bug with BTRFS’s implementation, as well as cheaper raid cards which don’t include a battery backup or other persistence mechanism. All parity setups do not have write hole issues. Poorly designed parity setups have write hole issues. Higher grade raid cards and ZFS don’t have this issue.

[@postnick]

I need to buy a PCI expansion card as my old system only has 2 SATA 6 and 4 Sata 3 ports. My goal is to just run exclusvily SSD's in my unraid, but I'm not sure if it knos how to deal with no HDD.

[@ashishpatel350]

FIIIBBBBBBEEERRRR

[@leo11877]

Is this recommended for large 50-90GB single files?

[@patrickprafke4894]

You missed sripped and mirrored raid 5's and 6's.

[@henri470x]

Can we please have tutorial of how to make your RPi Wireless KVM? Thank you :)

[@neilquinn]

I thought newer 10Gbit stuff was supposed to be a lot more power efficient now with the move to smaller lithography?

[@WolfgangsChannel]

Yep, but DAC is still more power efficient than RJ45 since you don’t need a transceiver

[@roysigurdkarlsbakk3842]

What's wrong with mdraid? You also have lvmcache, not tiering, but still

[@Ender_Wiggin]

The problem i don't seem to understand is that Unraid does not move things that are accessed frequently to the Cache..? I have been running unraid on my server for quite a while and it never moves frequently access things back to the cache just stores the things that were most recently moved to share in the cache. The implication about how you were talking about tired cache is that things are moved back and forth dynamically and that has not been the case for me. I could have my setup messed up and would love hear that. Also am really wanting them to be able to add more then one cache array. The fact i cant have to cache arrays for torrents and another for editing is annoying.

[@WolfgangsChannel]

Install "CA Mover Tuning". Also, you can absolutely have multiple cache arrays. With the caveat that each of them will have to be on its own set f drives

[@Ender_Wiggin]

​@@WolfgangsChannel But you can't use those cache arrays for anything but the Main array. I can't have a cache array for my ZFS pool. This defeats the purpose in my case. Also, the Unraid cache prefer does not balance recently opened files. Please take a look at other posts.

[@WolfgangsChannel]

Correct, it doesn’t do it automatically to that extent. What it does is take files that have access dates older than a certain value, e.g. 14 days, and moves them between the primary and the secondary array on a schedule. If you want a more granular and immediate caching behavior, ZFS+L2Arc is probably your best bet. And yes, you can only have a “cache” array for your main HDD array. That’s unfortunate

[@scottstamm7022]

Is there any point to a caching drive for spinning disk....if you have a dedicated RAID controller w/ battery and onboard cache?!

[@omarthatha]

Hey Wolfgang, any recommendation/advice on building a home lab but outdoor? e.g. balcony? what should I be looking for and what should I consider?

[@WolfgangsChannel]

Depends on how the weather is where you live. I personally don’t have any experience with that, but Mikrotik has some “rugged” outdoor switches, and I’m pretty sure that you get an “outdoor”/embedded server or even just a PC case. Still, that will only protect your electronics to a certain extent, and if it’s very humid or rains a lot, that might be a problem

[@omarthatha]

@@WolfgangsChannel Appreciate your time answering buddy, I am located in Estonia xD, and currently it's -11 and I was thinking of putting my whole rack into a DIY Wooden-isolated box with a few fans to control humidity, but a bit terrifying lol

[@WolfgangsChannel]

Yeah, I wouldn’t do it unless you have rugged equipment with a high enough IP rating. Which isn’t cheap

[@omarthatha]

@@WolfgangsChannel yeah, I am leaning more on the skeptical side of the moon with the outdoor setup, alrighty, time to do it Wolfgang style :), have a good one

[@robertt9342]

What happens if the cache drive fails, is what is “preferred” on it lost? For example, if I set my docker Plex app to preferred, and may ssd dies, is that plex and it’s library gone?

[@WolfgangsChannel]

Not if your SSD pool is redundant