Tyan Transport CX: Perfect for VMware Clustering! Our build with Epyc 7443

The music cuts out at a certain point because Premiere was being Premiere when it rendered this video. Sorry guys!
-Grant
forum.level1te...
Warning: FTT = 2 not recommended for smaller numbers of hosts!
**********************************
Thanks for watching our videos! If you want more, check us out online at the following places:
Website: level1techs.com/
Forums: forum.level1tec...
Store: store.level1tec...
Patreon: / level1
KoFi: ko-fi.com/leve...
L1 Twitter: / level1techs
L1 Facebook: / level1techs
L1/PGP Streaming: / teampgp
Wendell Twitter: / tekwendell
Ryan Twitter: / pgpryan
Krista Twitter: / kreestuh
Business Inquiries/Brand Integrations: [e-mail]
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to [e-mail].
-------------------------------------------------------------------------------------------------------------
Intro and Outro Music By: Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons...

[@Mallchad]

I know hardly anything about vmware of server chassis but somehow I found this video format exceedingly entertaining.

[@CarlosGarcia-tw8yw]

Idk what 90% of this video spoke about, yet I'm engaged.

[@NomenNescio99]

Now, this sounds all great until the time comes to pay VMware license fees.

[@BoatMurderedDF]

Paddle out to deep water? 😅

[@ytytiuiu2590]

Enterprise Plus - Acceleration Kit $24,250

[@Waynome]

@@ytytiuiu2590 You forgot the mandatory support that adds another 6k

[@TheKev507]

Love to see the shout out for not using USB/SD for boot anymore, that's been a real thorn in my side. Great video!

[@TheThenewdiabolic]

I'm a VMware admin and definitely would love to see more VMware content. Always looking to expand my knowledge.

[@tylerlindberg7881]

I'd love to see more advanced vmware stuff, and some automation stuff with it. :-) I'm currently running a two node vmware cluster with vRealize, and Horizon. I'd love to see more!

[@Casper042]

So many issues with this video. Like how exactly do you use a vSAN Policy to put a copy of a given VM on every node? Not sure you can, you can only adjust the FTT and let it take care of the rest. "You can't have an 8 node system with copies in 2 different racks" Yes you can, it's commonly used for Building Level redundancy in larger clusters and is known as a Stretch Cluster. Similarly, vSAN has support, for several versions now, Multi Tiered Failure Domains. So you can configure Rack A and Rack B as higher level domains, and then each Node (or in a larger config each Chassis) as a second level Domain. vSAN will then do what it can to spread copies over both racks and within a rack over multiple chassis if they exist. In Stretch mode at least 1 whole copy is in each Rack/DC, so it costs more disk but it's potentially way more redundant. I know this isn't meant to be a vSAN primer, but for those thinking of taking this and running with it, I beg of you to do more research and reading because some of this is a bit misleading, albeit I'm sure unintentionally. Cormac Hogan is a great vSAN resource from VMware's own Tech Marketing Team. Other things that caught my attention: - FTT 2 requires 6 nodes, meanwhile the paper on the screen shows 5 nodes. Keep in mind without an external witness, 1 node always acts as a witness or tie breaker. So even RAID 1 + FTT1 = requires 3 nodes minimum, 4 for auto recovery after a node loss, or 2+Witness if you were doing ROBO for example. - vSAN on 2 Nodes with your traditional SAN as a Witness = wrong, the Witness is a VM and can live anywhere, it's more important it be on a different HOST. What disk you store that VM on doesn't matter. You could arguably get by with a NUC for your Witness (Though I would still check the HCL) Also 1 final note, VMware Caps the Socket License at 32 cores. So if you buy a 64core AMD Proc, you will actually need 2 socket licenses per Transport CX node. So it's potentially good power savings, but a Dual 32 Core wouldn't cost that much more because the licensing and support ends up being a bigger overall factor in your TCO than the 2nd processor and more DIMMs.

[@Level1Techs]

I said ftt2;requires 6 nodes unless that was cut. And two chassis is ideal because of cross rack stuff but if you lose a rack the cluster will fail because you'll lose 4 nodes. What I was trying to say on ftt2 was that other competing clustering solutions don't have a problem replicating a vm across all nodes if that's your defined storage policy. Raid1 equiv storage policy would happily sync that vm to every node. I was surprised vsan didn't support this since open source clustering stuff does that just fine. For the two node witness I was talking about running the witness on a Synology Nas. I'm sure that's not recommended even though you said it can run anywhere ;) Stretch is for trans datacenter not trans rack as far as I understand. I don't think we really disagree on anything. I didn't want to get into stretch clusters as a way of dealing with failure domains which would still I suppose be ftt1? If you did keep the stretch cluster in the same DC Ftt2 is possible cross rack in the same non stretched cluster but idk if anyone would actually do that

[@MusikMan8675309]

Take a shot everytime Wendell says "Storage policy"

[@Level1Techs]

"hyperconverged"

[@MusikMan8675309]

@@Level1Techs We need to synergize and touch base after putting that in the parking lot

[@pabss3193]

Hi Wendell, where can we purchase these servers from? Been googling for a link to buy, pricing or anything like that and so far empty handed. Thanks!

[@creker1]

Like someone else said, you should really do more research before making a video about such systems. Many mistakes and misleading statements that will definitely cause problems when people start following this video 5:50 the number of drives in a node doesn't matter. You can have a single drive in capacity tier and still run all the storage policies you want. What matters is the number of nodes because every copy will live on a separate host to achieve the specified fault tolerance. Maybe you meant "with four nodes"? 6:30 running nodes without storage is nothing special. With recent vsphere you don't even need that host to be a part of vsan cluster because vsan can be mounted remotely now. All done in a couple of clicks and works completely transparently. We recently deployed mac mini that way to be able to run macos from remote vsan cluster. 8:15 there's nothing wrong with 2 node vsan. It works exactly the same as 3 node because FTT=1 storage policy is 2 replicas+witness. From FTT point of view, 2 and 3 nodes are equivalent. What you get with 3 is more capacity and compute, nothing more. 9:00 the reason you can't run FTT=2 on a 4 node system is simple. FTT=2 means 3 replicas + 2 witness components. You can't possibly run that on any less than 5 nodes. And you don't need 6 nodes like you said. Documentation says minimum 5 nodes. Also, replicating on all nodes is meaningless. What happens when you add another node? All VMs start replicating on it? What if you lose a node? Does that copy just disappear and vsan doesn't care or should it warn you that your VM is degreaded? That's just stupid. FTT is a reliable and predictable way of measuring your cluster resilience. 10:00 well, you can survive loosing an entire rack. If all your VM components live in a single chassis then it doesn't care if another chassis fails. You can't properly setup failure domains per rack anyway as vsan requires minimum 3 domains or at least 2 domains and a witness in a stretched cluster. 11:15 you can't replicate across all nodes. Vsan doesn't work that way. With 4 nodes you can only do FTT1. With FTT1 and RAID1 it means 2 copies and a witness component. Meaning VM will physically live on 2 nodes and the third being a tie breaker. With FTT1 and RAID5 your VM will be striped across 4 nodes and no more. FTT1 is still your fault tolerance and you can't go higher than that. The fact that RAID1 and RAID5 use different data placement doesn't affect VM resiliency at all.

[@Level1Techs]

So you're saying with a two node cluster it's fine because the witness can run somewhere else but with ftt2 you couldn't run a second witness outside a node adding vsan capacity? How is it not exactly the same scenario?

[@creker1]

@@Level1Techs I don’t really understand what you mean. Second witness? Is that even possible in vmware? Only deployments I know of use single witness. Like 2 node or stretched cluster. Everything else stores witness components on storage nodes.

[@thomasesr]

People gotta stop using standard MicroSD cards for hypervisors, there is such a thing as a Industrial grade MicroSD cards, they are more expensive, but have even ECC and are way more reliable

[@andrewjohnston359]

The only esxi USB drive that has ever died on me was the HP sold and sanctioned enterprise grade one for their proliant - awesome, thanks HP. I'm with VMware on this change of tune

[@garyadams7913]

What about a TrueNAS Enterprise or even 2TrueNAS Core as storage backend for redundant storage? With all the redundant compute would not ZFS be the optimal choice for a data store?

[@kanguruster]

Optane NOT on an Intel platform?! I thought that was forbidden?

[@draconightwalker4964]

id love a video on SANs and Fibre Channel stuff, Wendell

[@glennsteen]

Is that plain vSwitches, not a dvSwitch? Or are my eyes betraying me?

[@AdmV0rl0n]

Observation is both in kit, network, and tech/software levels, its being draw up into very rarified places. We are really heading into a lot of stuff being managed by fewer and fewer - yet very high level skill required peoples...

[@kemmow25]

SO MUCH MONEY, but ya pretty cool.

[@memyself879]

You cN tolerate losing 4 nodes on an 8 nodes setup using a stretched cluster configuration

[@Level1Techs]

True, yeah, didn't think about that

[@kennethmadsen6474]

"Pleb tier" 😂

[@amrnassef5344]

What video is that from @ 6:25

[@FlaxTheSeedOne]

this would be great if it would be available to purchase. having a hard time finding anything from tyan in Germany and europe for that matter. I hope they will work on availability and may even sell themself and not through resellers.

[@FlaxTheSeedOne]

@asdrubale bisanzio honestly if this is even liek 12k its a bargain for a business looking to buy this for a colocation and migrate away from AWS. Even for a small business this would repay itself in like 6 months

[@FlaxTheSeedOne]

@asdrubale bisanzio I know I was just talking chassi price.

[@alphanimal]

VMware, not VMWare 🤓

[@Ehren8879]

Ewww, Dell switches

[@bw_merlin]

Does the operating system have to be installed onto a separate drive outside of the tiered storage pool? Can the OS drive be part of the pool at all?

[@creker1]

It has to be separate drive. Sata Dom is great for that. Fast like ssd and small like usb drive.

[@JackBandicootsBunker]

Considering the prices for EPYC Milan, and the features these are amazing value for money as a 2U system.

[@Level1Techs]

Epyc ,third gen 24c Milan really is a steal in price/performance for common workloads

[@Hadw1n]

Man this really motivates me to study more.

[@MichaelRodriguez-rf9sc]

Yes yes and yes👍

[@vxzulu0o0o]

Hello 2014.

[@Feed9Will]

Wow - nice server. Hey now - 2 node robo is cool! Licensing of VSAN tho 👎. Did you give it a go with local 10G NICs between 2 node and witness traffic on MGT VMK? That is an advantage - can make vmition + VSAN storage traffic happen 10G direct connect without need of 10G switch. Other big consideration is how it is used as stretch cluster. Each node being in fault domain, one can Disable "Site read locality" so VM I/O goes to both nodes and this great improves performance. Of course bad idea with slow WAN links / separate faults domains. The main selling point of the 2 node robo though with witness appliance at a Central office kinda presents another decent investment...HA firewalls + 2 ISPs.

[@Gryfang451]

Thanks Wendell for another awesome video! Been using VMWARE for years. Still salty about VFlash. But, I have recently moved my backups to large Synology Rack units. Best decision ever. Active Backup for Business is so nice compared to Backup Exec or Netbackup. I'm able to recover whole VMs in a short time, individual files almost instantly, etc. My offsite backups are done with Hyper Backup, between two places. No more getting in the car with tapes, driving to the other office, storing said tapes, or even trying to do something even hoakier... Loving synology for this!

[@betohfinger879]

so awesome

[@GeneralCondom]

Me likey

[@JoshLiechty]

I can't wait to be able to have one of these servers in my home lab in 10 years! Seriously, though, I'm interested to learn more about the Synology UC3200 - the dual controllers in a product at this price point make me wonder about things like how well it handles failover, and whether it can really update the OS on the controllers non-disruptively. I look forward to your analysis!

[@Deveyus]

Now, replicated that without using VMWare.

[@QuentinStephens]

I wonder how much of that you can do in Proxmox?

[@andrewjohnston359]

Yeah I've done this with proxmox and old spare servers.. Configured the network the same as Wendell's suggestion as well. Would be nice to have the funds for this setup though. What would be nice is some tutorials on setting up effective tiered storage in ceph...

[@creker1]

@asdrubale bisanzio "ceph documentation and tutorials are so much better than vmware documentation for vSAN" that's because vsan can be setup in a couple of clicks. It's impossible to understand how ceph works and how to deploy it without reading tons of documentation

[@creker1]

@@andrewjohnston359 cache tiering in ceph has been long deprecated and not supported

[@creker1]

Probably everything but it all comes down to stability and maturity. Proxmox is no where near vmware stack in that regard

[@creker1]

@asdrubale bisanzio it is deprecated and been removed long time ago from documentation of red hat ceph. If you tried searching you would get many advices from the community to stay away from it. Even ceph documentation says that most workloads would run worse. If you skim through their github you would also see mentions of deprecation and attempts to remove it from the codebase. It doesn't work properly in ceph, deal with that. Setup proper cluster with ssd and hdd pools if you need hdd that badly. NFS or fiber? Good joke. It's not the 90s anymore. If your host crap out your VM migrates to a different one and components are restored on a different node. That's what FTT number is for. Yes, vmware is stability and that's why they're the leader in the industry when it comes to hyperconverged solutions.

[@DMSparky]

Lol I have literally know idea what’s going on in this video but I enjoy watching it because Wendell.

[@thatLion01]

Amazing video. I wish my work would buy one of these puppies

[@Na0uta]

Ya know when the white border comes out, its getting real!

[@johnpaulsen1849]

Have you done any testing with compute only nodes?

[@MrSiddhartha99]

I have commented earlier also.. those are the blade servers ,chassis is a blade server chasis.. this kind of chasis is not new and companies are using blades chasis for VMware infra for quite a some time..personally I am working in Blade chasis for last 5 years.. Dell Vrtx chasis. Dell M630 ,M640 Blades, Cisco UCS B-200 baldes ..these are common blades Server infra currently

[@Level1Techs]

Yep we reviewed the dell blade enter a while back. These are shared nothing however in terms of infrastructure. And each unit has an accessible pair of pcie slots. Fairly unique in "blades" and more standard imho

[@Meatguitarhamjam]

Compose. Adapt. Overcome.

[@soup3ygnome173]

Loved this video, I am trying to setup a small home lab with vmware for the purpose of learning automation tools as well as vmware because it is interesting. One day it will be up and running and I am excited for it

[@Level1Techs]

Best of luck! ~ Editor Amber

[@b2bb]

if only I were more of a DevOps guy

[@nikolaj5054]

Love these videos

[@isithaumayanga5687]

👍👍සුපිරි (Super ) this youtube channel

[@amarvir7444]

Wendell on 2x speed is great! :D

[@TedTabaka]

QNAP is for business. Synology is for home use.

[@Casper042]

@asdrubale bisanzio Or HPE or Dell/EMC or some of the newer Distributed Storage platforms. But yeah in general agree, I laughed when I saw the Synology mention in the video.

[@NomenNescio99]

I do consulting for a company that provides analytical software for Hitachi Fibre channel storage, they use synology for all their own storage.

[@Casper042]

@@NomenNescio99 Unfortunately the industry doesn't matter, it's the size of the environment that usually does. If this company only has 50 employees, I can maybe better understand that decision. Most of my customers are in the 10s of thousands of employees and none of them use Synology or Qnap. It's kind of like that Verge PC building episode. Just because you can use a Swiss Army Knife to build a PC, doesn't mean it's the best tool for the job, especially not if you build PCs all day long.

[@I4getTings]

Cool discussion! On those Dell switches, you'd probably want to run them in VLT (Virtual Link Trunking) so that you can have them present a single PortChannel to each server and then an additional PortChannel in 100gig up to the spine switch. Individual sessions are going to get hashed just to one switch or the other, but across a lot of sessions they will balance across the two Top of Rack VLT switches.