Inside a 32x100GbE Switch and its Big Flaw

STH Main Site article: www.servethehome.com/inside-a...
STH Merch on Teespring: the-sth-merch-shop.myteesprin...
STH Top 5 Weekly Newsletter: eepurl.com/dryM09
We check out the Celestica Seastone DX010 32x 100GbE switch and look at the different components inside and out. This is a popular switch platform for learning SONiC open networking. It is based on the Broadcom BCM56960 / StrataXGS / Tomahawk platform which means it has a lot of great L3 features. We then discuss the potentially major flaw that can brick this switch and many like it.
--------------------------------------------------------------------
Other STH Articles Mentioned in this Video
--------------------------------------------------------------------
- Get started with 40GbE SDN with Microsoft Azure SONiC for under $1K (Arista): www.servethehome.com/get-star...
- Intel Atom C2000 AVR54 bug released but hidden by NDAs: www.servethehome.com/intel-at...
- Intel Atom C2000 AVR54 bug prompts C0 stepping: www.servethehome.com/intel-at...
- The first time the AVR54 bug struck STH: www.servethehome.com/intel-at...

[@bork3592]

I got one of these and minecraft is smoother than ever

[@bushhawk5460]

I though: "Hey, that's a neat price" until I realised that you are going to spend at least twice that ammount on transcievers to populate the ports.

[@ServeTheHomeVideo]

DACs are pretty cheap for QSFP28. The QSFP56 ones for 200GbE are still expensive. I think a lot of people are using these with DACs which keeps power low and costs way down.

[@GuruEvi]

Twice? A branded 100G SR transceiver is basically the same cost as the switch, hell Cisco will sell you a brand new switch for this cost, given you buy their transceivers. Even second hand the good transceivers will end up going for $300+, not sure if you want to trust transceivers that are lower cost in a critical setup.

[@ServeTheHomeVideo]

@GuruEvi I do not think a $1000-1500 4+ year old switch like this is going to be using top-shelf optics. 😀

[@bushhawk5460]

@@GuruEvi or run in a critical setup

[@JosephHalder]

@@GuruEvi $29 for a 1m DAC, $39 for a 2m DAC on FS.com. No reason to use optics for a top-of-rack deployment. That being said, their SR4 transceivers are only $99 if you need to reach to other IDFs.

[@henrik2117]

2020: 2.5 Gbit becoming more common in retail computer motherboards 2016:.... 🤔

[@wildmanjeff42]

Thanks for the video Patrick, Love STH channel and content. Learned a lot from your channel :)

[@jozefaz]

We got hit by the intel chip bug and had to RMA over 2000 Cisco 4K routers. What a nightmare that was.

[@ServeTheHomeVideo]

For those of us who have seen it in action, it was not pleasant. Some folks have not seen it and are unaware/ do not believe in it. We even get comments to that effect now (see comments on this post www.servethehome.com/inside-a-celestica-seastone-dx010-32x-100gbe-switch/) but I still think it is important to raise awareness. 2000 routers! That stinks.

[@spx2327]

How can you not love this guy!

[@TheBackyardChemist]

If you are handy with a soldering iron, and you can find the correct pin/trace adding the resistor mod should be possible.

[@critical2210]

Man I'm just happy with my cheap gigabit netgear switch lmao

[@squelchedotter]

I'll be very happy when that tier of switch finally gets to 2.5 and 5 gig :)

[@JuMPeRaBG]

You got that switch for a $1000? This is insane!!!

[@ewenchan1239]

I got my 36 port 100 Gbps Infiniband switch for just under $2300 USD. (Retail is at LEAST $11k.) This is part of the reason why despite the cost of adapters, cabling, and the switch itself, on a $/Gbps basis, it’s still VASTLY cheaper than most other speed tiers.

[@michaltatka]

Thank you for the review - I bought two, shortly I will test it (and SONiC sound interesting too). Time to strengthen the team Cisco 3064PQ (48x10GbE+4*40GbE) and 3164Q (64x40GbE), used for LAN.

[@zipp4everyone263]

Man, this is cool! The bug isn't but the hardware is really really cool! I love the look of hardware and knowing what this is capable of just makes it so much more interesting! Thank you for the rare insight into this piece of DC hardware!

[@excitedbox5705]

Is that the bug where the Internal clock dies? You only need to add a new resistor/solder to another resistor. Dave from EEVblog had a NAS with that CPU and he shows how to fix it.

[@Jonas_Meyer]

Yes it is. He mentioned the resistor fix/rework in the video at around 7:33. Not sure how exposed the pin would be on this switch through.

[@KrisLipscombe]

I really like that placing on the power supplies, that’s just a nice feature and shows a bit of thought. Completely agree that you’d expect some better user servicing access at this price, particularly when this kind of switch is going to be key infrastructure

[@someguy4915]

You'd expect exactly less user serviceability for these, typically you won't even replace the fans or PSUs when they break, the manufacturer gets the call and replaces them for you. You don't pay them for nice equipment, you pay them for things to work, on cheaper servers I've had to replace a few broken HDDs but the blade systems just got taken care of by HPE and Dell if anything broke on them, too expensive to do it yourself.

[@nateb4630]

How interesting! This is way outside my wheelhouse these days, but I used to play with big SONET gear, and lately I'm doing a lot more hardware hacking. While this had some gorgeous views, it left me hungry for more detail! I'd love to see more perspectives from different disciplines in here, especially a more detailed look at the board-stack connectors, all those debug headers all over, and the various silicon aside from the Broadcom piece de resistance. Also, how does that heatsink work being so wide; is there an integrated heatpipe or something? This hunk of hardware has more stories to tell!

[@steven44799]

The bright yellow battery holder is likely because they were cheaper than a black one, in a product like this where it's unlikely any customer is going to open the product it makes sense to just go with the weird colour part to save a bit on BOM cost as nobody buying them will care what colour it is.

[@johndododoe1411]

Depends if battery lifetime is less than service contract duration. Then there would be service instructions for customer staff changing batteries during maintenance windows, and then having a bright colour on the only user serviceable part makes sense. Besides, 12 screws is not as user unfriendly as the 4 non-removable screws I found when decommissioning a Gb D-link switch from 2010 and wanting to find and destroy the config flash chip before recycling.

[@ion337]

Nice teardown guys. If feasible it would've been interesting to see the underside and board-board interconnect for the switching PCB. Guessing it's regular PCIe between the BCM ASIC and x86 CPU?

[@TheMschipp]

Yep, normally PCIe 2.0 x1 for that generation

[@PrestonBannister]

Interesting, as I really do not have a handle on the white-box network gear market. My use case is a high-end special-purpose gadget that moves a lot data. Kind'a sort'a looks like a datacenter in a box. (What I would *really* like to find is an *unmanaged* 25GbE switch, oddly.)

[@NvidiaTVH7]

I would love to see a video on a low cost sonic network os setup, maybe a future video idea?

[@ServeTheHomeVideo]

We had the article by Chris linked in the description for this on a 40GbE network. It is pretty similar to other switches. I actually think that kind of CLI heavier content is better on the website.

[@johndododoe1411]

Is there a good alternative not from a company known to be hoarding data from "customers"? So obviously not Facebook / Google / Microsoft / AT&T.

[@ws2940]

Great video on this switch. Thank you for making it. Definitely looks like a Cisco clone of some sort.

[@paulstubbs2778]

Interesting, the bit about switch efficiency & cost per gigabit. Around here I am still mainly on a 100mb switch, it uses way less power and my internet connection cannot saturate it. If I am mucking about, and want to move a ton of data, I can easily patch in something a lot faster, however as a daily driver, watching ServeTheHome video's etc., the old Alloy 100mb is it.

[@thatLion01]

I'm surprised this 100gb switch is already 4 years old. I never actually seen 100gb used at any of the dc I sit. I did a few 25/40gbe upgrades that cost a fortune

[@johndododoe1411]

4 years ago, 100GbE was mostly for backbone providers and Internet exchanges, not datacenter internal networking.)

[@thatLion01]

@@johndododoe1411 thanks for clearing my vision. I never realized that! Excellent point 👍

[@MK-xc9to]

Well , this switch is far to oversized for my needs , the Intel Atom C2000 would be another " NO GO " . For my private usage i use the Mikrotik CSS 326 with 2 x 10 Gbit SFP+ and 24 one Gbit ports , passiv = silent and max 20 watts ( i believe ) for 130 Euro . I only need High Speed to my NAS , for the rest like TV , BD player etc is one Gbit enough

[@nilswegner2881]

I'm kinda mad that I only just upgraded to gigabit a little over a year ago. Now there's switches that do 100 gigabit for under 1000 euros... Thats so insane, I was totally fine with my 100 mbit equipment before. I mean sure, gigabit brought a hughe improvement, file server wise, but my internet connection still is only 50 mbit so I still have some devices on the old 100 mbit link. :x

[@MRooodddvvv]

I totally need one of those for my crappy 4G 1 megabit at good weather internet !

[@Jellman86]

Love your content, are people actually serving the home with this class of equipment? Would love some fresh homelab content. Looking to refresh my stuff, what to buy in 2021?

[@ServeTheHomeVideo]

Totally. These are quiet enough that if you have a basement rack they are very usable. We have had people using these at home in the forums for over a year. On the home lab side, really Xeon Scalable/ EPYC is about the best there is aside from the TinyMiniMicro series we are doing. Intel has some new Atom and Ice Lake-D coming, but those have been delayed a bit.

[@steven44799]

One of the network guys we use at work has a cisco 9500 series switch in his home rack. Even he admits it's complete overkill and unnecessary, but he wanted it anyway.

[@Jellman86]

@@steven44799 @servethehome I can beleve it, If I had the money is be all about this. Hard one to get past the wife!

[@jfbeam]

@@steven44799 I have a small collection of Nexus kit as well, but it's still sitting around. They're too loud, and eat too much power to run them 24/7.

[@ewenchan1239]

@@Jellman86 I’ve deployed 100 gigabit Infiniband in my basement now. I was able to get it passed the wife because I used to have a CAE on consulting business where I was running HPC applications where stuff like this REALLY came in handy, and the Infiniband switch, even moreso. Start your own business and tell the wife it’s a business expense.

[@thibaultmol]

That thing looks so similar to the dell switch that Linus looked at recently. So does dell use this company oem as well?

[@draconightwalker4964]

I’m thinking it’s the next model up the stack

[@ewenchan1239]

I don’t think so. Linus’ switch is 48 25 GbE SFP28 ports with 6 100 GbE QSFP28 ports. This is 32 100 GbE QSFP28 ports. Total switching capacity is higher here.

[@Jonas_Meyer]

Maybe they used the yellow battery holder for servicing. I suspect this would be the only component that actually get serviced so the technician can identify it faster. Or maybe they used them on another project or got them cheap.

[@Fuzzyfull]

Did you test the power with all QSFP modules installed? As I understand, the large PSUs are for powering the transceivers.

[@Phil-D83]

And I am happy with 4x 10gbe ports on my.home network. Lol

[@maxvideodrome4215]

I’m good with 1G ports :)

[@hariranormal5584]

maxvideodrome Same. Modem ==>> Wifi router. End of my network

[@bluegizmo1983]

Currently in June 2022 you can find this switch on eBay for less than $500

[@Sbellins1109]

WOW 1000dollar for a switch even if it is 100GB for me it's too much ahahha. I've bought a couple of days a go a Juniper EX3300 (48 1gb RJ45 ports and 4 SFP+ 10G ports that can also compatible with 1G SFP) for 110€. It works like a charm (for now only at 1gb because i need the cards) and it have a very nice web interface. I hope that this Juniper switch is not picki with the transcivers and does not require it's own brand for 10G, for 1G it works with off brand once)

[@JosephHalder]

I'm using fs.com and 10GTek DACs with my Juniper EX2300, it also has 4x SFP+ ports. I don't believe it will be picky. It might throw some minor events in logs, but it works fine. Mine are "Generic", you can get fs.com and others that show as branded "Juniper" to the switch for no additional cost.

[@zactodd3144]

Point of interest. Is the c2000 resistor fix something that could be done to a switch like this. Have carried it out on a synology unit myself and Patrick did mention the fix, but sounded like it was implemented at the manufacturer level.

[@michaltatka]

It could be fix by resistor ? (on the CPU PCB board?)

[@zactodd3144]

@@michaltatka yeah, C2000 series Intel well known for the issue. Got a $5 DSJ415+ (company threw away as it couldn't be trusted). Running 2 years before died, and now running a year since fix.

[@michaltatka]

​@@zactodd3144 Wow :) do you have any links how to fix it? waiting for mine switch, and if it's a 15 minutes it is worth it (if my model have the cpu bug). and anyway I will open it to switch fans to "PC RGB ;)" to make it silent like we done it with Cisco 3064PQ.

[@michaltatka]

@@zactodd3144 ok received, mine is 2017/2018 :) no resistor needed.

[@zactodd3144]

@@michaltatka good job. Yeah, the resistor fix is all over the web/KZbin for the Synology NAS... I was just asking if anyone knew of a fix for the switches. Good to hear you got a later model one where the issues had already been fixed.

[@AaronR-C]

@ServeTheHome I'd love to hear some suggestions for other cheaply available switches in the multi-100Gb port range (using the same Broadcom chip as this, or equivalent) that aren't running on the C2000 platform. You mentioned that they exist?

[@ServeTheHomeVideo]

Have one inbound via FedEx ground.

[@AaronR-C]

@@ServeTheHomeVideo A) You're awesome B) Looking at this, and the STH article about SONiC on the Arista box led me to the SONiC supported hardware Wiki... and though it doesn't list which CPU a switch uses, it certainly seems like a great jumping off point for more research. github.com/Azure/SONiC/wiki/Supported-Devices-and-Platforms Thanks again for all you do.

[@Dragirek]

**Linus want to know your server location**

[@ayuchanayuko]

Too late, LTT just posted a review of a 48x switch ;)

[@Dragirek]

@@ayuchanayuko Terabyte Switch master race

[@tnaxpw]

@@ayuchanayuko wasn't that 4-6 100GbE and rest at 25GbE?

[@nilswegner2881]

@@tnaxpw yup because 100 GbE Nics are still not affordable nor good value

[@justjoeblow420]

I wonder how hardcore the re-work is to address the bug and if it can be bodged onto an existing switch with a steady hand and some know how. If I had the money to build a proper compute cluster like I would love to do right now this would be a good candidate switch wise outside of the bug with the processor.

[@eDoc2020]

It shouldn't be too hard. I don't know the specifics of this switch but the LPC bus is often brought out to a 2.54mm pin header. You'd just need to add a single resistor to this header. On EEVblog 1288 Dave did the repair (and explained the problem) on a Synology DS415+.

[@CCL13CN]

They probably just purchased a huge amount of CR2032 mount for some legit reason, and just keep using them for all the products they have decades down the road.

[@miaudottk9080]

Get someone to replace that CPU. With a Zhuomao rework station it's possible. See where you can find a CPU and contact Rossmann Repair Group and ask for the rework price. They have the station, but I have no idea if they're willing to do the job, because they repair macbooks, not servers.

[@craigcordeiro8530]

I have no money and no use case but I want it.

[@KuntalGhosh]

Lol and i was thinking 10gbe switch is too expensive and unnecessary 😂

[@jfbeam]

They certainly used to be. If you can live with optics and/or DACs, 10G gear isn't too insanely expensive these days. (and "n-base" (less than 10G) stuff is getting cheaper) "Unnecessary" is still a matter of degree -- most don't need it, but it can be handy if it's there.

[@velcroman98]

This would be great for setting up very large computers. The type that consist of racks and racks of blades that make up a single computer.

[@KuntalGhosh]

@@owowowdhxbxgakwlcybwxsimcwx sfp is a pain the ass for home network. The cables r expensive.

[@jfbeam]

@@owowowdhxbxgakwlcybwxsimcwx In every way :-)

[@someguy4915]

@@KuntalGhosh SFP is just the socket, second hand transceivers can be found for ~$10-15 and in a home network you won't need hundreds of them. Fiber cabling second hand or new on FS.com is also pretty much the same cost as decent quality CAT6/CAT6a cabling, just DACs are typically kinda expensive...

[@NUCLEARARMAMENT]

I can only get the 1.5 Tbps IB routers.

[@michaelloving8004]

I'm lookin @ buying this switch seller says it was manufactured 2019 will be safe from that bug ?

[@beauregardslim1914]

So much bandwidth on old gear like this and yet major brands are still shipping 1Gbit ports on new motherboards, WiFi 6 APs, etc.

[@everythingfeline7367]

When the vast majority of internet connections don't get near gigabit (and often not even 100M), there's no good reason for the extra cost for most people

[@LtdJorge]

@@everythingfeline7367 for LAN

[@clausdk6299]

Maybe... it's time to upgrade from 10gbe to 100gbe at home 🤔 🏠

[@Thorsummoner0]

i really hate how behind the times my 10gbe home lan is :)

[@marksapollo]

100GB for a thousand bucks!! Hell most home networking enthusiasts are busy installing CAT 7 and 10GB networks. And from the sounds of it these are peanuts compare to what the enterprise market uses now. Amazing.

[@ServeTheHomeVideo]

This type of 32x100GbE switch is still ultra-popular. You can use breakout cables and hit 20x 2U 4-node servers on 25GbE with these switches and still have plenty of uplink bandwidth with a bit of over subscription.

[@wiziek]

No one should be installing cat7, cat6a is fine or just use fiber. Cat8 cables will be super short anyway,.

[@marksapollo]

@@wiziek And who is going to spend the time installing fibre network cables around there homes with all the hassle they give, over Ethernet Cat 7 cables?

[@marksapollo]

@@ServeTheHomeVideo Amazing, thank you for the video.

[@veegee24]

When you say "not come back up", do you mean permanently fail? Or does it just need a cold boot?

[@ServeTheHomeVideo]

AVR54 is beyond just a fold boot. It can usually be repaired via soldering, but it is not a simple fix like swapping a fan/ PSU.

[@UpcraftConsulting]

Looks VERY similar to my Dell S5232F-ON. I bet Dell is using these guys as their OEM. In fact I see Linus at LTT just did a video with internal shots on his 25gig version and the internals look extremely close. kzbin.info/www/bejne/Z2nbpaKdn99ohdk The Dell s5200 series appear to have different switch controller (the pcb below toward the back) Likely Dell using their own design here and it would make sense since they bought Force10 and use that OS10 operating system so they are likely doing their own thing on the control side. A linkedin profile of a Dell employee definitely shows Dell, Sonic OS, and Celestica joint venture as keywords in the profile preview but I'm not logging in or paying linkedin to view the complete thing to dig further. Kind of confirms that Dell s5200 series is basically either an identical or very close offshoot of these. Not surprising though, Dell often takes some generic ODM design often customizing software or at least packaging it as a solution with other dell equipment to spin up a lot of their enterprise product lines.

[@charlesselrachski34]

100 giga-bits? Makes me feel like marty in 1955. Doc all I needs is a little 2.5 giga-bit ports.... kzbin.info/www/bejne/nF6aaKurodB1lbc Remember this moment, We'll be looking back in 2255(when we can haz 100 giga-bit ports) and feel like that video - looking in to the past.

[@TacoGrande007]

Why are all these high end switches so cheap on Ebay? What is the catch?

[@robertkelly5391]

This generation is at the replacement age. newer tech is faster and more efficient.

[@robertkelly5391]

If my switch has a manufacture date in 2018 I should be alright, right? I got mine for $450

[@ServeTheHomeVideo]

Wow nice! I would suspect that has a fix already.

[@tylerlindberg7881]

I have two AG9032 V1 Agema Delta 100gbe switches, but I cannot find any way to load an OS on them. :-( Have you played with these switches?

[@ServeTheHomeVideo]

Not yet :-/

[@chuck2501]

looks like the inside of the switch on LTT recently.

[@ServeTheHomeVideo]

This is a bit of a higher-end unit.

[@chuck2501]

@@ServeTheHomeVideo yes, the spec is way greater but layout is very similar, I was supposing Dell are one of those purchasers.

[@chrisb945]

this looks much like a dell switch on the inside... in the end... they all seem to be manufactured by a single vendor ^^

[@ewenchan1239]

Mellanox, I think, had this type of switches in like 2015.

[@AlfaPro1337]

Will Linus benefit from this?

[@ServeTheHomeVideo]

I know they are deploying 100GbE/ 25GbE right now, but I think they are using a Dell switch.

[@DJ-Manuel]

Yes, they use a dell switch, but maybe in a few monthes they releas a video „25Gbit wasnt enought“ 😅

[@ServeTheHomeVideo]

@@DJ-Manuel I mean... I know they've gotten 100GbE running just from chatting with them.

[@DJ-Manuel]

Yes, the newest video on floatplane shows off their switch, but from what i can understand they use 100GbE mainly for server connectivity and not for client, not sure if the OM3 cables they‘ve deployed (seen in some youtube story of them) would even be capable of 100GbE, my guess would be that this is the limiting factor for them to do 100GbE as well to the clients (and of course, at the moment they don‘t need it) 😅

[@wiziek]

Linus doesn't really knows anything about networking to be honest.

[@opiniondiscarded6650]

Bruh do a video on how to DIY rework it

[@martinum4]

Maybe ask Louis Rossmann^^

[@MrCadilLACI]

can you stack these for mc-lagg?

[@johndododoe1411]

The backbone chip can only do 3200Gb, so at 32 100Gb ports, it's maxed out and can't handle any stacking traffic beyond what can be done via the ports.

[@georgewright1093]

Why is the name of this "ServeTheHome"? Are people using any of this at home?

[@ServeTheHomeVideo]

George, absolutely. A switch like this that lets someone learn SONiC on relevant hardware is extremely useful to those who are in the networking space. To your broader point, STH is now almost 12 years old so we have expanded a lot. There is a brief history on the STH KZbin 1 min channel overview. While STH KZbin is small, it is only a low single digit percentage of traffic we get across STH with the vast majority being on the STH main site. It is kind of like how the Wall Street Journal expanded beyond covering just what happens on a street in New York City.

[@joeldoxtator9804]

Interesting, but F.Y.I. you will never need this in your house. These are made for network backplane racks that connect multiple server rack rows together. The only way you could ever come close to saturating the bandwidth of this switch is with multiple racks of server all communicating at once over 10GbE. But then again, there is no kill like overkill.

[@MrAtomUniverse]

Can i dry an egg on top of it?

[@Gastell0]

I guess you better not do a port mirroring on this

[@jimmyyu2184]

I dunno, Rocky III wasn't that good, and Rocky IV, and then Rocky V, Ugh... Sylvester S. kinda gotten old by then.

[@ServeTheHomeVideo]

Ha! Great.

[@majstealth]

*cough* 32x100GbE being outdated, we still deploy 1GbE because, "costs"

[@calaphos]

Seems wasteful that you cant just replace the simpler management mainboard with that CPU while keeping the (presumably) expensive switching board.

[@user-km7vz2xh2m]

0:20 left board from a cheap laptop

[@rodfer5406]

Holy sh.... !

[@dashtesla]

Linus = :/

[@tannertalan1209]

Hi Guys 😍💋 💝💖