Inside an Arista 32x 100GbE switch that had a $30k List Price

STH Main Site Article: www.servethehome.com/inside-t...
STH Merch on Teespring: the-sth-merch-shop.myteesprin...
STH Top 5 Weekly Newsletter: eepurl.com/dryM09
We take a look inside the Arista DCS-7060CX-32S which is a 32-port 100GbE switch based on Broadcom Tomahawk silicon that can run SONiC. This is also an interesting switch since it has an x86 processor, namely an AMD embedded chip so as a fun fact, it technically has AMD GPU IP in it as well. This switch was around $1000/ port list price new which puts the list price at around $30K when it was first launched. Since then, street pricing new is somewhere in the $9000-$12000 range these days and on the second-hand market (if you want to use one with SONiC to learn on) these often sell for $2000-$3500.
--------------------------------------------------------------------
Other STH Articles Mentioned in this Video
--------------------------------------------------------------------
- Celestica Seastone 32x 100GbE switch teardown: • Inside a 32x100GbE Swi...
- Edgecore AS7712-32X 32x 100GbE bare metal switch teardown: • Inside an $11K Edgecor...
- Hands-on with Intel's co-packaged optics demo: • Hands-on with Intel Co...
- Inside a Supermicro-Intel 48x 100Gbps Omni-Path Switch: • Inside a Supermicro In...
- Get started with 40GbE SDN with Microsoft Azure SONiC for under $1K (Arista): www.servethehome.com/get-star...
- Intel Atom C2000 AVR54 bug released but hidden by NDAs: www.servethehome.com/intel-at...
- Intel Atom C2000 AVR54 bug prompts C0 stepping: www.servethehome.com/intel-at...
- The first time the AVR54 bug struck STH: www.servethehome.com/intel-at...

[@fabianlibert7910]

One way to remember red vs blue fans would be to think of it as the temperature of the air traveling through the fan. Blue draws cool air in, red draws warm air out. @ServeTheHome

[@ServeTheHomeVideo]

I like that one! You would think that since blue is my favorite color I would gravitate to blue fan units.

[@joshuaboniface]

Was also thinking "Red goes in hot aisle, blue goes in cold aisle" as a way to remember that. Much better colour coding than some products (glares at Cisco).

[@Felix-ve9hs]

This Switch is so fast, it can even make you coffee and do your tax declaration

[@jeremybarber2837]

Uncompressed 4K video over IP is definitely driving 100G networking in my field.

[@post-leftluddite]

Shut up

[@UpcraftConsulting]

We got 2 of the dell branded Celestica 32 port 100gig switch last summer. Dell list price was also crazy high but they discount it down a TON. I think the high list price is 100% because the maintenance renewal is based on list pricing not your discount pricing. So after year 3 or year 5 depending on what warranty you got to start with your renewal price is based on the very inflated pricing. (Usually 15-20% of the original list pricing to do renewals is the forumula)

[@seangreenberg3212]

Easy way to remember Arista airflow is the color is the temperature you should feel on that side, red is hot (exhaust), blue is cold (intake).

[@jozefaz]

We got hit by the c2000 bug 2 years ago and had to RMA 1800 Cisco 4K routers in the field in production. And also last month in my lab a 4331 Cisco router died on building power loss.

[@tincoandringa4630]

You always get my like just for pronouncing the model numbers.

[@ServeTheHomeVideo]

The hardest part of these videos (by far!)

[@ElijahPerrin80]

Fantastic, I really enjoy your experience and views.

[@calvinteh3297]

This switch costs more than my car.

[@ServeTheHomeVideo]

List price $30K means street prices are in the $9-12K range. Used prices are much lower (I put this all in the description.)

[@Damicske]

That's more than all my cars that I've owned :+ (9 currently)

[@SupremeRuleroftheWorld]

@@ServeTheHomeVideo so, it costs more then my car....

[@GrahamCantin]

My 1996 Honda Civic cost me $2400. My DCS-7060CX-32S cost me $1875+shipping. Costs less than *MY* car. :D

[@thomassvedin8701]

The faucet for hot water is red and cold is blue. At least here in Sweden. So if the handles are red your hand will be hot when you come near.

[@pmsrodrigues]

In my case the Atom C2000 hardware failure hit both my pfSense firewalls, even after the manufacturers BIOS update. Fortunately the failure was one year apart, and they were in a HA configuration.

[@spx2327]

Patrick has it all! He is handsome, knowledgeable and eloquent. How can you not love this guy!

[@That_Stealth_Guy]

I totally want the heatsink. It's just begging for a 100W LED attached to make an insane desk lamp! It would look so much better than then one I have on my old GPU heatsink.

[@bassman87]

the best way to remember which color is what is to think hot isle/cold isle. blue is intake, red is exhaust.

[@bassman87]

oh man I remember that atom bug. A bunch of our customers were hit by it. If I remember correctly it was around 120 days that people were realizing they bricked their device.

[@anthonydiiorio]

Wow what a monster.

[@excitedbox5705]

The C2000 bug is a problem with the realtime clock. If you solder a resistor to it you can fix it. There is a lot of money to be made buying "broken" NAS and networking gear and replacing a 1 cent resistor and selling it. Dave from EEVblog has a video of him fixing a NAS. You can get the broken gear for a couple hundred and sell it for a few thousand once fixed.

[@ewenchan1239]

I kind of wished that you would have plugged it in and let it initialize and then stabilize so that I can get a sense as to how loud this is compared to my Mellanox MSB-7890 36-port 4x EDR 100 Gbps Infiniband switch that I already have. It would be nice for me to drop in 100 GbE into my network given that my Mellanox ConnectX-4 cards are dual port VPI, which means that I can run stuff on the IB port AND also run stuff on the ETH port as well. Thanks.

[@berndeckenfels]

Compared to a lowly Force10 (24xSFP+) for 16k this was a deal

[@alfredopreciadomolina1576]

The blue version of the fans is for manufacturing and test purposes and the airflow is inverted because the red ones can not cool the chips without the top cover.

[@castlers5075]

I bought one of the t shirts but after a wash or two the logo is beginning to peel :(

[@LarryTheRoleplayerTM]

That's why you don't buy shitty youtuber clothes.

[@user-dq4sh2mt8l]

@@LarryTheRoleplayerTM ahahaha

[@Razor2048]

Any chance for the switch, they will release a coupon code that offsets the gouging, and lowers the pricing to around $200-300 for the switch, including a few QSFP modules?

[@Shadowauratechno]

If it has a gpu, there must be some way to game on it... 🤔

[@stonegray230]

If you're brave you could wire up that NVMe port to a full-size GPU...

[@FelipeQueirolo]

"switch running doom with activity lights"

[@jamess1787]

Install SONIC, steal/borrow/acquire/compile a patched version of X.org, and BAM: gaming machine. Might have to pipe /dev/video over /dev/ttys0 ... Somehow.... OR Might be limited to using a MATROX graphics card... (Once you figure out how to attach a USB PCI adapter card to the board).

[@eDoc2020]

@@jamess1787 Couldn't you just pipe the graphics output through the network? I'm not sure how quickly the SoC could pipe packets into the switch chip but it should be usable at least.

[@jamess1787]

@@eDoc2020 of course. But I was thinking retro doom. Hah.

[@soniclab-cnc]

Yep... my Asrock Rack mini died from the AVR bug. The board was replaced on warranty and I also bought a spare board. This is for my main NAS and I cannot have downtime... spare board on the shelf just in case it dies again. lol

[@excitedbox5705]

You can fix the C2000 problem by soldering on a resistor. Check out EEVblog video where he fixes a Nas with the same problem. It is a problem with the realtime clock.

[@jdbiphone]

Any experience or reviews of FS.Com 100G switches?

[@HomelabExtreme]

I don't know i'f i'm blind, but where is the OS stored? The older Arista switches used to have a 2GB eUSB, which i can see pads for on the PCB but it is unpopulated. The never ones have M.2, but that is not installed? Did you remove the m.2 for this video?

[@lost4468yt]

I'm fairly sure the Atom C2000 bug *is* something which will end up effecting every chip sooner or later? It's not something that might effect some chips as far as I know? It's something that will effect all chips eventually. E.g. it's pretty much impossible to find an original Xenon Xbox 360 because they just all died in a rather short period of time.

[@Primoz.r]

Does red on the handles mean hot side and blue the cold side?

[@AWPneeson]

Samz should pick this up

[@samegoi]

Thx

[@davidwilkerson1904]

To let you know, if you are using an Nvidia graphics card with RTX technology, it does have an amazing filtering technology which automatically cuts out background noise and can do it live streaming. There are several KZbin videos displaying what seems to be filtering on an impossible level but until you experience it yourself you tend to think it is some kind of hoax. Pretty sure that is what happened with the leaf blowers.

[@ServeTheHomeVideo]

It turns out that having a studio with fancy mics and sound dampening all around did this as well. :-)

[@jolness1]

32x100gb for 2-2.5k was unthinkable a few years ago. Very cool.

[@wiziek]

There are switches coming out with familiar amount of 400g ports. Big webscales are refreshing their devices.

[@jolness1]

@@wiziek I love how quickly stuff moves in data center/enterprise applications. Being able to snag hardware that is a few years old for a fraction of the cost is great. Especially considering what my needs are. I look forward to when 7002 series epic are EOL. I know that there isn't the market saturation like with intel but still, will be cool to pick up massive thread count CPUs for so much cheaper than retail

[@popcorny007]

@@jolness1 There's a big difference between hyper-scalers and your generic enterprise IT department :P One needs all the speed it can possibly get, one is happy with a 5 year server lifetime

[@peterpain6625]

@@jolness1 Considering some migration plans i've seen 7XX2s should be out on the used market Q3/21-ish i guess. In volume Q4/21

[@jolness1]

@@popcorny007 That is true but enterprise hardware still adds even more supply to incredibly cheap older hardware. But yes, they are different, hyper-scalers switch hardware out like I do oil in my car.

[@ABUNDANCEandBEYONDATHLETE]

In 2011 I was building 100Gb networking in my college st. Network project. 10 years ago. How much is this? Sent as of 1 min In.

[@pr0ntab]

$1k a port when 100gb was new is frankly a bargain. We routinely paid that much for 48 port 40gb (stackable) switches just a few years later.

[@ws2940]

Im idly wondering how much it would cost to fully equip this device with all the bells and whistles OS and software wise.

[@WesFelter]

EOS is free but back in the day BGP was $7K, ZTP was $7K, and automation was $7K or something like that. Of course you negotiate discounts on licenses.

[@Webtroter]

When you wanna plull the fan, do you want to feel hot air or cold air? If you associate blue with cold, you shouldn't mess up :D

[@therealb888]

Enthusiastic network noob here, how does this compare to the latest gen 100GbE switch? I'd assume newer ones have lower latencies & lower power consumption but by how much? Any other new cool stuff on the newer gen switches?

[@ServeTheHomeVideo]

A really easy one is that more modern switches are 400GbE that can break out into 4x 100GbE for higher density.

[@therealb888]

@@ServeTheHomeVideo Wow that's amazing, they can reach some insane densities with that.

[@wiziek]

@@therealb888 Not only switches but routers. Check this out, 32x400 gb ports in 1u. With bigger chassis routers you get 36x400g in one line card (because CPU and network ports are seperate components for chassis like thse). www.cisco.com/c/en/us/products/collateral/routers/8000-series-routers/datasheet-c78-742571.html

[@fbnx4219]

What would be the reason to put an SSD and extra memory in such a system? I mean it is a switch and probably provides all functionality in the spec sheet as it is?

[@d00dEEE]

I could see putting an SSD on it, if you want to do local logging.

[@tarzan12345]

I got one of these for my home network. 😆

[@mattw3406]

Can I plug this into my Asus 3100 router? ;-)

[@Damicske]

Yes use a SFP with max 1Gbps connection, more is idioticly :)

[@wiziek]

@@Damicske I'm not sure those support sfp copper 1g ports, may work only with 1g fiber sfp. That would be pointless anyway.

[@leocelente]

But can you run Doom on that GPU?

[@woldemunster9244]

If Skyrim runs on a fridge and DOOM runs on a ATM....

[@TheSillymansam]

Can't wait to see linus drop this!

[@Phil-D83]

And I am content with 10gbe for now...lol

[@jschenard]

I have one at home for my gigabit internet. It's just fast and awesome 😅👌😂

[@danielius2924]

Why the hell do I watch these, I barely know how to adopt a unifi USG without breaking everything

[@user-zs2bh9ul2c]

Can i use this as a normal switch, dumb switch

[@TheMcSebi]

but, can it run crysis?

[@jamess1787]

You'd have to ask the 2 guys who are outside going at it....

[@zipp4everyone263]

The constant "Why are we improving X? Back in my days we didnt even have X!" comments are really annoying. Why do we have cameras that can shoot 8k? Because we open up for new markets, we can, but also because the professional industry wants it. Why do we use 100Gb switches? Because we can, we open up for new technologies and strategies and because the market demands it. If we'd never upgrade our stuff we'd still be running around with asbestos cigarett filters, ozone killing spray bottles, sub 1Mb/s internet and just generally be much worse off for it. Sometimes it sounds as if the internet is full of Amish people. Which is incredibly funny if you think about it...

[@ServeTheHomeVideo]

What is funny is that when we started installing 100GbE years ago in the STH lab I kept getting asked "why?" and I basically said it is less expensive than 25GbE switches and I do not want to have to rip and replace 40GbE in a few years.

[@zipp4everyone263]

@@ServeTheHomeVideo see! Not everyone thinks about the work and replacement cost of re-installing the same stuff over and over again. Sure, you can't really future proof tech, that's always going to be a loosing game, but you can do estimations of where some tech might be in the future. 100Gb networks apparently made it :) Can't wait for the day when 100MegaBytes are the new standard speeds for our wan connections. Imagine what we could do with single point 10Gb connections and low latency infrastructure... I mean just the fact that routers today are more powerful than my first computer is insane, I can't wait for the day where we don't have to choose between LL or IDS.

[@Veyron640]

just talks too much..

[@NatesRandomVideo]

I really do like your videos but an awful lot of stuff is hideous overkill for serving “the home”. Stuff needs a separate channel, honestly.

[@ServeTheHomeVideo]

I am totally confused here. STH is almost 12 years old and we have been covering this type of gear for more than a decade. The Wall Street Journal covers more than a road in NYC. STH YT is newer/ and very small but it will cover the same topics that we do on the main site.

[@NatesRandomVideo]

@@ServeTheHomeVideo certainly. I just don't understand the "home" branding for stuff like this. No biggie.

[@popcorny007]

Don't read into the name "home" too much, it has always been about high quality enterprise hardware reviews. From the home to the data center.

[@GrahamCantin]

My set of 800GB SAS SSDs begs to differ. My home is full of enterprise gear I got off ebay for cheapcheapcheap. Some of it is crap, like my foundry switches with built in accton backdoors, some of it is absolutely worth it's weight in gold (my quad opteron workstation can still compete with epycs for another year or so, and it was made in 2009...) Why buy western digital spinners when you can get Micron P420m 1.4TB PCIe SSDs? I literally ran out of PCIe slots and have been pondering getting a dual Epyc board just for the 64 PCIe lanes. Oh. Oh, I get it. You probably don't live in silicon valley. That's alright too.

[@NatesRandomVideo]

@@GrahamCantin Not sure what living in an overpriced area has to do with tech at home, but... whatever... thanks for a list of your 12 year old stuff... also have no idea what it had to do with my question about the branding... (rolls eyes)...