Review: How is That 45 Drives Storinator Petabyte Project Going 7 Months Later?

[@timmitchell9021]

One of the reasons I love Tom's content: "Build for your use case."

[@45Drives]

Thank you Tom, Thank you very much 🤝

[@malice930]

Storinator has been my go to for the last several years (XL60 and C8). As veeam backup respo's and virtualization storage (10GB iscsi freenas). They have been rock solid. Keep it simple. Great support and warranty.

[@malice930]

@@franciscooteiza why, never had any issues and performs well.

[@rdwatson]

Thanks for the update. This is definitely a KISS situation, FreeNAS/ZFS is a very reliable solution if you don't go cheap or add unnecessary complexity on top of it.

[@JzJad]

Simplicity is best when possible, doing ceph single node would have been pointless, and if need be they could go for scale later at this point

[@marcogenovesi8570]

Ceph single node is just weird. If you don't need a proper ceph cluster, ZFS is the way.

[@nadtz]

"Only" 200tb of data. Haha most of the clients I deal with would have their jaws drop to hear that number. Nice to see the follow up!

[@Furriee]

Hello Lawrence. I admin several 45 pods for several years now. I think, the first one we started around 2014. Total size 2+ PT plus backups :-). And no, no Ceph. Cheers mate.

[@michaelbeaver4804]

I am a Storage Analyst, I am currently running a 10 Petabyte Solid State Storage solution on 40 Gig Nessus Switches for redundancy. The whole system is constantly mirrored to a AWS node.

[@mitcHELLOworld]

Sounds cool, are you using any type of SDS solution to cluster that into a single namespace? Are you using it as File system, object or block? Would love to hear more about what a 10PiB flash solution is doing, and also man that must be some hefty AWS bills

[@mitcHELLOworld]

Dun dun dunnn

[@michaelbeaver4804]

@@mitcHELLOworld file system for medical records

[@mitcHELLOworld]

@@michaelbeaver4804 yikes. That is the type of thing I would not want to know about being in the cloud.

[@mrfrenzy.]

@@mitcHELLOworld Then you need to make sure your local (or federal) politicians have a clue and stipulate high data protection and segregation, for example like GDPR. Unfortunately no organisations or companies will do extra work to protect data unless they are forced.

[@Mike_Hartman]

I feel like needing a petabyte of storage but not needing many, many 9s of uptime is a pretty rare requirement. Cool project though and 45 Drives seems like a good solution for it.

[@marcogenovesi8570]

It's a thing in research/R&D, or also in media production where the users of the data are mostly internal and can handle waiting some more hours or days for the result of their elaborations to come in.

[@KevinSatterthwaite]

the company i work for fits that exactly. many of our employees generate 100's of GB of data weekly. That data gets stored for 10 to 15 years and sometimes only needs to be accessed once or twice during its lifetime. But when it is, it needs to be nearly instantly. Not a few hours later. We are sitting around 700 TB right now. 1 PB is not too far in the future.

[@PupShepardRubberized]

wags wags, My technology life has improved so much when I built my first nas in 2016. Network storage is something that most people could benefit from.

[@Traumatree]

Im curious as to what SLOG devices you used for that project ? When you build the server, you choose the 60 x 18TB drives, but no write-cache drive. What do they offer and what did you use?

[@LAWRENCESYSTEMS]

Did not need a SLOG

[@MrXuegui]

Anyone have a good breakdown of 45drives versus other solutions (ie Supermicro, EMC, Dell etc)?

[@Jordan-hz1wr]

When you picked "FreeNAS" is that and older version of FreeNas? Or will it ship with proper "TrueNAS"?

[@mikeh4327]

really the big question is how you handle the backups for these type of databases. i'd appreciate something about that in details. i have few clients that run about 2-5TB. and though i can back it up. what scares me is that the recovery will take forever to get the data back and also when i reinitiate the backup it can take forever to do the first backup. thanks in advance!

[@LAWRENCESYSTEMS]

We are replicating the data to another local storage device (2 of them) that are physically taken to another location on regular rotation. They did not want to transfer data to or from the cloud.

[@noyou5870]

What is the power draw for that system?

[@hariranormal5584]

Wow, even a "cheap S3" service takes 71k$/year for 1PB, a machine with 1PB ish is just 41k$? wow

[@ws2940]

very cool. Thank you for the video.

[@kenzieduckmoo]

Hey, i know this might be too vague of a concept to do, but often i see people talk about a "virtual machine storage target" could you make a video explaining what that IS practically? Like in my mind im imagining a corporate citrix style layout where a user logs in and it spins up a VM for their session, and thats running on one machine, but the "local storage" for that VM is this storage server in a completely different machine, which to me seems WAY too slow to be useful in business, but maybe its exactly that?

[@morosis82]

That's pretty much what he's talking about. Speed depends on a few things obviously, but if your storage is spinning rust, hiding it behind a fast network interface isn't going to negatively impact it. A good reason to do this is failover, so a VM can be spun up on another host basically instantly if it's host dies or is shutdown with no need to copy it's data. If the target dataset is large enough, this is pretty much the only way to do it. In my lab I don't have full failover, but I have all my bulk data on one server and am moving toward a distributed solution for the nvme disks the VMs run off (ceph). The VM runs on a local disk with replicas in other machines, but any larger datasets live on a single machine with big disks. That data is not mission critical stuff, so I don't mind if I have to shut it down occasionally for maintenance.

[@marcogenovesi8570]

Yes, that's exactly what "virtual machine storage target" is. A dual-controller storage appliance with a realtively large drive array, possibly with SSDs. Netapp, Nexenta, Oracle, Dell/HP/IBM/Lenovo and a bunch of others make that kind of device. And no it's not slow at all, that's what 10Gbit/25Gbit/40Gbit/whatever network connections (or fiber channel connections) are for

[@oleksandrlytvyn532]

Hello, I just installed PopOS 22.04 (first time for me as Windows user), I noticed that there is no review for PopOS 22.04 yet and maybe something changed in Shortcuts. Would be cool to watch video from you on that topic :-)

[@MR-vj8dn]

If you want to start small, can you add drives over the years to come?

[@LAWRENCESYSTEMS]

VDEV's in ZFS have to be built symmetrically so there is a way it can be done.

[@MR-vj8dn]

@@LAWRENCESYSTEMS Thanks. Also: Are VDEVs and ZFS the only path with TrueNAS?

[@LAWRENCESYSTEMS]

yes

[@marcogenovesi8570]

@@MR-vj8dn You aren't limited to TrueNAS on this hardware though, there is also UnRAID where you can add drives one at a time like that

[@MR-vj8dn]

@@marcogenovesi8570 Thanks. Sounds like I need to have a look at UnRAID as it sound more flexible. My storage needs change over time as customers come and go. Even customers have different needs.

[@xmurisfurderx]

You can build a similar setup for less than 10k

[@peachycaper]

Good Cape Breton company!

[@Panda-ek9ll]

Is the company openAI?

[@ewenchan1239]

It's too bad that 45Drives doesn't offer a 100 Gbps add-on NIC option.

[@LAWRENCESYSTEMS]

They do if you call for custom build.

[@mitcHELLOworld]

we definitely do :)

[@ewenchan1239]

@@LAWRENCESYSTEMS Good to know. Thank you.

[@ewenchan1239]

@@mitcHELLOworld That would be awesome. I wished that you could also pick (on the online/instant quote system) HGST/WD Ultrastar drives instead of only Seagate/Toshiba. I'm also curious as to how the hard drives fair with the NVH coming from 60 drives at the same time.

[@marcogenovesi8570]

It's literally a supermicro motherboard in a semi-cusomt chassis, you can install whatever NIC you want, or buy it without drives and put whatever drives you want in it

[@bentheguru4986]

Still has same issues of way too expensive.

[@mitcHELLOworld]

yeah, a petabyte of storage isn't super cheap. Who knew right? 😂

[@TechySpeaking]

first