I just wanted a cheap Epyc server...

Grab yourself a Rocks Glass and set of Whiskey Stones at craftcomputing... for when your BIOS updates fail.
So much wasted potential. The saga of the Quanta T22HF Dual-Node Epyc server somehow continues, as a couple viewers of the channel managed to get their hands on the BIOS updates needed to bring Epyc Rome support to these boards. And somehow even the simplest task for this server still managed to go sideways.
But first... What am I drinking???
Highland Park 12... because every failed BIOS update needs Scotch.
If you have a Quanta T22HF and want to try this update yourself, BIOS files and Documentation can be found here: drive.google.c...
Links to items below may be affiliate links for which I may be compensated
Think you can do better? Here's the Quanta T22HF with Epyc 7351P CPUs and rails: ebay.us/je8UK6
Follow me on Mastodon @[e-mail]
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing

[@VEC7ORlt]

Wow, that merch plug at the very beginning - have a downvote.

[@CraftComputing]

Wow, this video posted for you to watch for free about multi-thousand dollar projects might want to keep being able to do it. It was 37 seconds of a 24 minute video with no 3rd party sponsors.

[@marcogenovesi8570]

wow that salty comment at his short plug about his in-house quality merch - have a pin of shame

[@bluesquadron593]

Wow2, you can always just go away and not come back.

[@VEC7ORlt]

@@CraftComputing yeah YT without sponsorblock is just craptastic.

[@derpsakry4464]

silence, redditor.

[@AgencyNighthawk]

This is a great indication for me to never recommend Quanta to my clients.

[@hubertnnn]

At this point we are running out of companies really fast.

[@dh2032]

when basically telling you to bin the product, and buy a new one? at what was it, 10,000 thousand, something price tag? they basic saying you are not rich enough company for our products?, and you know your right where not?, time do our shopping else where? where we are rich enough to spent our cash? ¯\_(ツ)_/¯ 🙂

[@RobinCernyMitSuffix]

I use some older (E5 v3/v4 era) Quanta servers, they work, but yeah, no support, no manuals, no bios or BMC updates. _Don't use quanta_ just stick to AsRock Rack, Gigabyte and Supermicro servers.

[@JosephHalder]

Quanta is really in the business of selling thousands of units at once to large vendors, as well as being an ODM for other large manufacturers. In either case they just don't have public facing support. It's just not how they're structured. It's a shame, but I understand why they do the business that way. It would step on the toes of the actual sellers of their rebranded equipment. I don't disagree with the sentiment though.

[@RobinCernyMitSuffix]

@@JosephHalder They wouldn't have to provide support, all that people are asking for is to be able to download documentation and firmware for their old products.

[@praecorloth]

19:50 "... buying a product for which the original manufacturer refuses to acknowledge it ever even existed." Cisco has entered the chat.

[@anothersiguy]

I will say from my limited experience reviving a C220 M4 it’s been easy to find support docs and BIOS updates. Having a Cisco support login from work has definitely helped though. Plenty of Cisco proprietary jankiness, but doesn’t sound like they’re nearly as bad as Quanta.

[@stevewalton2959]

Meraki, the deceivingly cheap ebay listings that probably get oh so many people that have no idea they need a license.

[@Prophes0r]

@@stevewalton2959 Or Brocade. Luckily in that case there is a reasonably easy to find community of dedicated "external support people" that have solved most of the problems already.

[@aklinko]

Liebert pats Cisco on the back and takes its seat in the chat. Had a lead-lag system for a pair of 20-ton Liebert CRACs that no longer exists according to Liebert's customer service, despite the thing being mounted to my wall.

[@francescogugliuzza3827]

NetApp: *heavy breathing*

[@JosephHalder]

HP will acknowledge a server, even give you a changelog on the drivers and BIOS... wait, you want to download them? That will be $2000 for a year of support.

[@Prophes0r]

Customer: Yeah so I have this Quanta server. Can I have the documentation and BIOS for it please? Quanta: I'm sorry. What server? Customer: Model QuantaX111 Quanta: I'm sorry but we don't make that. Customer: You, Quanta, don't make the QuantaX111? It came out 5 years ago. It has your logo on the front panel and your branding on the BIOS. Quanta: No we don't make that. Customer: There is a 1 year old internet archive page of your website showing a product page for the QuantaX111. The latest BIOS update was 10 months ago. Quanta: No, I think you have the wrong Quanta. We don't make that one. Customer: I got this number from the support contract that I bought from you, that is still valid for another 4 months. Quanta: I don't know what to tell you. We certainly didn't make that server. But if you are interested in Quanta servers let me transfer you to our sales team... Customer: No wait....

[@tofailtowin9046]

i got sad reading this. fr

[@InfernalOd1n]

Gas lighting the public seems to be a corporate strategy these days.

[@Codeaholic1]

Well we are certainly showing that large numbers of us are willing to accept outright lies.

[@Prophes0r]

@@Codeaholic1 3am musing: It kinda makes you think about how the "rules" of our society only work when both sides are willing to follow them eh? Like, the systems we use only work when both sides actually agree to do the things they say, and actually do them. Like, If a company refuses to honor their side of an agreement, there isn't anything YOU can do about it. You could spend a bunch of time and money to take them to court, but if they really don't want to do a thing it is going to cost you a lot of resources. When things were "smaller" the system just sorta worked. A store sold to townspeople, and pissing locals off could wreck your business. Billion dollar companies do not have that risk. Some are big enough that you literally might not have a choice whether or not to deal with them if you want a specific type of product. That's the consequence of a global economy. Just some "Mind is winding down after a long day" thoughts.

[@silencer51]

I've made a habit, after quite a number of problematic BIOS updates from much more mainstream vendors, to always dump the known working bios to a file via the flashing utility or an SPI programmer, before updating... such a bummer :(

[@bluesquadron593]

But he can't flash the bios.

[@silencer51]

@@bluesquadron593 with a hardware programmer, even a cheap CH341A, you can flash pretty much all motherboard BIOS chips, usually without even removing the chip from the motherboard

[@Ramon200000]

@@bluesquadron593 Nothing a external flasher cant fix.

[@tacokoneko]

he needs a SOIC8 clip and a computer that has an SPI flash programmer port (like a BeagleBone Black) and he also needs the programming experience to use the flashrom program

[@silencer51]

@@tacokoneko in this case, the chip needs to be desoldered from the board (extra bummer). But it should be doable. The problem here is that, if there's no solid backup of the original BIOS image, apparently finding it is very difficult - the OEM certainly can't be trusted to provide it, sad as this sounds...

[@cocadmin]

Thanks for taking one for the team this time, hopefully next time will be the one :)

[@NamelessUnicorn]

Tiens, je savais pas que tu le suivais aussi ^^ (peut être lié à tes déboires avec ton serveur récemment ?)

[@cocadmin]

@@NamelessUnicorn haha oui je suis a fond dans les homelab depuis quelques mois. Les vieux epyc sont dans mon radar aussi :)

[@kieranwilliams3052]

Agree 100% for taking one for the team and being ao transparent to allow us fellow Home lab\Professionals to add this to our mental database to NEVER buy a Quanta piece of hardware! See Quanta fails to understand that some of us Home Lab freaks also have say with purchasing on our day job on purchasing of such new servers and it will be over my dead body on a Quanta approval to pass me now!

[@SpencerHHO]

Reach out to Rossman, Quantas lack of support may actually be illegal in some states like Cali with right 2 repair laws.

[@thecruzader4882]

Its a open compute project node sold only to select customers under NDA, anybody with a legitimate support claim has it available. By the contract it was sold under it cant be resold to others either. This is standard for the OCP hardware from all the large vendors, at best you have the specsheet on website.

[@ibex485]

If anyone has a bricked motherboard, router or other device which stores its firmware on a standard EEPROM, there's no need to buy an expensive dedicated EEPROM programmer. A Raspberry Pi can be used to read/write EEPROM chips, and does an excellet job. Just connect the right pins up to the chip and off you go. Sorry this project didn't work out for you Jeff, it was a noble attempt. Seeing enterprise equipment being treated as disposible after only a few years use is so upsetting, especially when it still has so much performance potential. Keep up the good work.

[@insu_na]

And that's why I only use SuperMicro, ASRock Rack, Gigabyte and HPE Servers. HPE of them is also shitty with their BIOS/BMC support, allowing you only to download them if you have an active support contract, but the others just give you everything you could want. The neat part about HPE is that even if they don't support homelab use-cases their hardware usually always just works. So far I've never had an issue with an HPE system that uses HPE certified components. Much nicer to use SuperMicro, ASrock and Gigabyte stuff tho, despite the jankiness of ASR and Gigabyte.

[@betawolfhd]

I'm enjoying my recent purchase of a Z640. Quite robust

[@ThirdEnvoqation]

If you work in the industry the server engineers are usually friendly enough to give you the necessary build files. It is how I got a 385 G7 back up and running with HPE software. you can also usually find the necessary tools on-line in server fora, that is reasonably safe to download I believe STH and LeveLOne can be a good source.

[@PsyMan2022]

Great video as always Jeff, As head Curator of the unofficial Bristol museum of enterprise computing in the UK (or shed full of junk as my wife calls it) I know first hand how our hobby can simply lead us to "throwing good money after bad" as they say. It's great to share these bad experiences so we can tick that idea off the list. Brilliant. I am currently sipping a UK Brewdog Triple Hazy Jane IPA at a rather cheek enpinkening 9.5 volts and will be also trying a scotch dismount in honor of your helpful video with a Talisker Skye I picked up last month when visiting the Northern parts of my island at their distillery, in my opinion the best part of the UK. Keep em coming.

[@unknowntotherestoftheworld]

I'm sorry for your loss.

[@jp-ny2pd]

Soon as you said investigate the BMC I was like, "Oh, oh no...". Cut to the scotch...

[@AutisticEnderman]

Thank you for posting this. We need a t-shirt for all of us who have been here. Hardware manufacturers who make stuff so proprietary that it ends up as e-waste rather than in a home lab or reused for education should be named, shamed and fined.

[@roadkill11000]

Learning about things that don't work, is as important as learning what does. Thanks for posting your fail video anyway! And yes AC is important as it is hot in the valley this week!

[@spazda_mx5]

If companies like this are going to deliberately make their old kit unusable by withdrawing docs, bios updates, etc. then they should have to take back the equipment at their cost for proper recycling, rather than getting to dump their now useless crap in the used market.

[@incandescentwithrage]

Most of this stuff comes from "free" IT recycling companies. They purge the data and sell either directly on a (differently named) website, or an eBay store. The techs usually don't care whether docs or bios' are available... it's still worth what someone is willing to pay for it.

[@PaulaXism]

And the marketplaces which facilitate the selling of this junk need to step up and protect their users who are being taken for a ride.

[@darkcloud7843]

The fact they have no BMC reset header is mind boggling to me. All other major brands have them for this reason. This way their techs don't have to RMA an entire server when a bios update fails and instead they can just reset the BMC, login to IPMI and then reflash the bios on site.

[@Prophes0r]

You gotta stop looking at this like a Tech/Engineer. If something like this confuses you, try this. Imagine you are an Executive. You don't have to understand anything other than how to make line go up. That is your only job. Now, every day someone uses something they already bought from you, they aren't giving you more money for new stuff. That is where support contracts come in. People paying you money without you having to design/make/store/ship stuff is GREAT. But, it isn't great forever. Eventually they need to buy new stuff, or you don't get to justify your quarterly bonus. Try to be as cynical and poisonous as you can. Doe it make sense now?

[@lauraprates8764]

​@@Prophes0r It's not logical either, since you ended support for the hardware and they're not likely to buy new hardware either and if they do buy new hardware they're less likely to buy from you. It's a LOSS-LOSS situation, so it makes less sense to give you bonuses when you're actually hurting the company's image and losing clients

[@BozesanVlad]

Chinese Lenovo RD450X has 4 dip switches to: 1. Stop BIOS write function, 2. restore ME, 3. Restore BIOS ("emergency BIOS recovery, used when BIOS code is damaged and cannot boot"), 4. start ME programming function Quanta seem better suited to fill landfills from factory :)

[@Prophes0r]

@@lauraprates8764 It is perfectly logical. Clearly you have never had to work with a purchasing department.

[@KSMcLeod75]

Thank you for posting the fail. In my journey into home lab and networking it’s a nice reminder that even those I follow and look up to run into the same issues I have. Now to start dropping hints to my wife for some new whiskey glasses 😄

[@FrenziedManbeast]

I saw lots of red flags from these kinds of proprietary servers when I was looking to upgrade to Epyc. Not to mention sketchy stuff like vendor-locked CPUs which I personally believe should be illegal. In the end, I found an Epyc 7313P for just over $400 and then paired it with a new ROMED8-2T and 256GB DDR4 3200. All in it cost me just under $1,400, but the build went together without any major setbacks. Now it's in an ATX chassis and not a slick rackmount node, but that was a compromise I was happy to make.

[@Prophes0r]

There is always the option of using a rackmount ATX case. If you look in he background of most HomeLab KZbinrs, especially the lower budget/newer ones, you will see a 4U Rosewill case like the the RSV-L4000U which can be had for $250. (RSV-R4000U is the shorter depth one) Sorry. While looking up the model number I saw there was an open-box one of these for $160 and I bought it immediately before finishing this reply...

[@FrenziedManbeast]

@@Prophes0r I looked into a few rackmount options including multiple Rosewill 4U chassis (they have at least two still listed on Newegg), but in the end I went with a used Rosewill Blackhawk: It had enough internal drive bays, good airflow, good clearances for my SP3 tower cooler, and enough external 5.25" drivebays to fit my three 8 x 2.5" SATA bays from Athena Power. I have 24 x SATA SSDs and several NMVe across M.2 and U.2 interfaces, so I wanted something to accommodate all that into one box. I might one day incorporate one or two small to mid-sized racks, but right now it's more of a nice to have.

[@varekb]

What a great episode! This hobby is not all rainbows and unicorn farts. While it's cool to get enterprise-grade hw for pennies on the dollar, there's always a risk. I love this hobby and think your channel is great. Thank you for "keeping it real" about the risks of used and refurbished hardware. Like most here, I've had mostly good luck but have definitely had a couple of duds over the years. Still worth it. Still great fun.

[@richardskinner6391]

The reason there are so many issues with Naples/Rome support on early Epyc servers is because vendors used 16mbit BIOS chips, as was common on early Zen motherboards, and there isn't enough room for all the AGESA versions etc. Vendors cut corners. The early versions of the HPE DL385 gen10 and the Dell r7425 have the same issue. Back then, AMD didn't have the market share to insist they use 32mbit chips.

[@ThirdEnvoqation]

No one wanted to invest in Zen 1, and AMD told them to use 32Mb modules (it was in their OEM spec design) but didn't have the clout to enforce it and hence why the OEMs cut corners and just shoved out the cheapest working solution they could and why OEMs do not like supporting hardware of this era. I still remember AMD boards with Intel 211 network chips which was approaching EOL being installed just to save money. The other factor was no-one expected AMD to support the socket for as long as they did because they gotten used to Intel changing their sockets at a drop of a hat.

[@Abrasive-Heat]

“And now I’m drinking scotch” is how several projects are gonna go sometimes 😂😂😂

[@peterpanozzo1123]

All companies are the same. I had an issue where a company refused to provide firmware for a device that still worked. On their website, they claim to be "green" and "environmentally friendly," but in reality, they're saying, "We can't profit from that device anymore, so buy a new one even if the old one still works. We get new money, and we don't care about the electronic waste."

[@mndlessdrwer]

Dell is one of the only companies I know where you can routinely download server firmware without needing a support contract with them. Their servers aren't as good as Cisco and their support isn't very good either, but they are at least that little bit more consumer-friendly. Cisco requires a support contract to download anything. HPe is the same. Most server manufacturers won't let you download anything without a support contract, which just seems like a bullshit anti-consumer tactic to me and I want the EU to do something about it.

[@RandarTheBarbarian]

@@mndlessdrwer Me with my dell last week, man. I reset it for a windows update and afterwards it would not post, neither did it give me a fault code. I stripped that thing down and had both CPUs and all the RAM out of it twice, nothing. I contacted the company that sold it to me saying I knew it was out of warranty and asked if there was anything they could do, and to their credit they had a guy go through everything on this system again and maybe a couple things I didn't think of. I swear it was a BIOS issue because 1) no post and no fault code with hardware that was working 3 minutes ago pretty much fits that bill and 2) when I was troubleshooting and doing research I found some claiming that that Dell system had issues initializing some non-dell graphics cards and 3) when I did eventually get it up after doing the same thing 3 times I updated the BIOS and it just seems to be normal again. My point being I went searching for a Dell BIOS for a 6 year old EOL professional system and it was still just there on their customer facing website, and PCSP decided to be cool and had someone try to help even though I was outside the warranty period for their ebay sales (they offer a longer included warranty if you get stuff direct from their website though, so I'll probably go there first when I get the next system).

[@terrorpup]

Whelp, I will join and have "Not your father's Root Beer", been there so many times Jeff. Sometime, walking way, working on something else helps, but for now. Cheers. Remember, it's why we home lab.

[@seylaw]

Sad to hear, mate. EPYC CPUs are also sometimes bundled to a specific motherboard vendor nowadays (due to a security fuse; or as I suppose, killing the secondary market on purpose). That should be indeed a crime.

[@CraftComputing]

Don't even get me started on vendor locking CPUs....

[@danub3r]

As a long time Quanta owner, I can say this tracks :-D. I love my LB6M though.

[@tradingnichols2255]

If I was you, I'd be beyond more frustrated than you. I've had some very harsh Epyc situations myself. I think you made it nicer because you wanted to keep your cool and share a video with us. As someone, on my other channel, who's uploaded a LOT of DIY for automotive... suck projects like this, I didn't always have the heart to post it. Bravo man! Job well done!

[@cute7752]

Did you update the BMC first, before updating the Bios? I think otherwise quanta doesn't like it.

[@user-xj8xw1mu4r]

A certain salty Luis Rossmannn might want a talk.

[@drakkon_sol]

Jeff, it's time to mount in on the wall, since you killed it.

[@gohamm34652]

Not sure this company is worthy of a spot on the wall of shame

[@tappy8741]

It's not dead, just a lot of hassle to get going again

[@jeremybarber2837]

Oh man… that’s super crappy. Thanks for putting the video out & posting your very eloquent, well reasoned opinion/experience. Wholeheartedly agree!

[@Blond501]

Hey Jeff, what could also be, that the BMC got incompatible with the BIOS. HP Enterprise made something like that with a Server I got, the Cloudline CL2200 G3 1211R. After some time I saw that this server supports Haswell and Broadwell CPUs, kind of odd thinking about the different architectures but well BIOS updates and so on ... And seeing that the Server had the same name suggested to me it could be supported. Great that I read the Documentation: It states Servers delivered with Broadwell only support Boradwell BIOS and vice versa. HP hat a even a big documenion website, that states the behavior flashing the wrong bios. That you're even able to burn the wrong bios blow my mind. But: When you flashed Broadwell BIOS on a Hashwell server, the BMC bricked! SO getting the server back online is only able via HP Support, which will now say: Buy a new one. So hearing you "Well I can update the bios via BMC all the time" made me shaking because of the documentation I read of HP. BTW: I read the documentation before applying any BIOS updates, so I didn't killed my server :D

[@JibunnoKage-cj2kz]

The problem also, is that IPMI is not always fully isolated from the server foot print. This situation can create issues and odd conflicts, comparable to what you discovered. Dell played this game with some models of their servers, which was really frustrating when Dell did not disclose this up front. Dell improved iDRAC to where it was directly comparable to HP iLO, so Dell addressed the issue, but if not for the extensive lab evaluation we did, the issue would not have been exposed up front as it was.

[@JamieStuff]

I've bought a few pieces of enterprise gear over the years. Whenever I see something I like, I go directly to the manufacturer's web site to see if they still have manuals, BIOS, and such available for download. I agree that it's reasonable that older gear no longer gets updates, but at least leave the support page up. That said, the last two server boards that I bought were both Supermicro...

[@shpyda]

I was just thinking of this server the other day. Thanks for the closure on this adventure! Hopefully the content is making up for the expenses and headaches. Thanks for always being informative and helpful in your content!!!

[@NikolaiCherepanov]

Gotta love the "you will own nothing and like it" mentality that these companies have.

[@Prophes0r]

This isn't even one of THOSE situations. This is willful intent to ignore the existence of a thing. Customer: Yeah so I have this Quanta server. Can I have the documentation and BIOS for it please? Quanta: I'm sorry. What server? Customer: Model QuantaX111 Quanta: I'm sorry but we don't make that. Customer: You, Quanta, don't make the QuantaX111? It came out 5 years ago. It has your logo on the front panel and your branding on the BIOS. Quanta: No we don't make that. Customer: There is a 1 year old internet archive page of your website showing a product page for the QuantaX111. The latest BIOS update was 10 months ago. Quanta: No, I think you have the wrong Quanta. We don't make that one. Customer: I got this number from the support contract that I bought from you, that is still valid for another 4 months. Quanta: I don't know what to tell you. We certainly didn't make that server. But if you are interested in Quanta servers let me transfer you to our sales team... Customer: No wait....

[@betawolfhd]

​@@Prophes0r it is. If you can't fix it, you have to buy something else and throw that out. What was the point of the fictional argument you had? Companies are destroying their documentation for old equipment to make the right to repair disappear

[@Eledore]

Thank you for the video Jef, and i am sorry that you might need to take out the server behind the barn. But this sparked a industry question of me that i will be forwarding to our procurement department. They to be contacting Quanta and request there recycle mandate/goals. And ask specifically how they are helping there customers meet ITAD and Enviroment goals. For us, Reuse is a higher goal then Recycle. We also have a non-discriminate policy. If Quanta is not helping all second hand users, then they as a product will not be compatible with our own Environmental Sustainability Goals.

[@bryce2113]

Last 5 or so minutes of this video is crucial and needs to be shared in the industry. Companies that intentionally create e-waste is so irresponsible. And to spend active efforts to thwart the secondary market from giving hardware new life is absurd. This is one reason I like purchasing old HPe equipment because they keep their files on their website and its easy enough to find. Shame on Quanta.

[@MrFroggyTech]

Maybe someone already posted, but for the ipmi login issue. Check the length of the password Field, it could be that the password gets truncated. It accepts a longer password but it just truncates it, so just a thought and a hail Mary 😁

[@kras_mazov]

That was very brave of you.

[@ChrisRoxby]

It might be worth mentioning, for those with similar setups, that flashing the BIOS reset the IPMI login back from whatever Jeff set it to.

[@zuexs690]

Jeff, thank you so much for posting your failures. Genuinely. It's incredibly hard to avoid impostor syndrome when every KZbin video you see is a clean, rehearsed version of the messy reality we live in. Some may have learned from this video the risks of homelabbing, but I learned that it takes a lot of guts to post something that didn't go to plan. Just for your bravery in posting this (and contrary to some... tasteful comments I've read so far) I'll be buying some whiskey stones and drink along with you as I stare at my piles of dead Supermicro project boards 🍻

[@demorez5]

honestly despite your struggles (or because of them) this was the most entertaining video i've seen on the homelab topic. it had a story, humour, suspense, glimpse of hope and then an important learning point. Also very relatable to my fight with a cheap a Fujitsu Xeon board that turned out to have a proprietary power connector using 11V instead of of 5 and requiring a specific external temperature sensor to POST.

[@dozerd42]

i appreciate being able to follow the successful projects you do AND the "failures." It puts projects like this into perspective. I put failure in quotes because we all did learn something about Quanta today 😀

[@supadupashooter]

Your statement about hobby with ex enterprise is On the mark. And this video shows my wife I'm not the only one going thru this. Hahaha

[@jamieficken2488]

Screw Quanta!! Thanks for taking the time to do this and for closing the loop on the project.

[@lumibraulius]

Good afternoon from Spain, What you indicate is one of the things that infuriates me most about "companies" (I work in IT). Quanta, NETAPP, etc.. what is the need to delete everything from the internet and leave your clients (we know yours is 2-handed) thrown away? It's just like they say around here: "Throw stones on your own roof"..... I love the channel and everything you do. Keep it up ;) ------------------------ Buenas tardes de España, Lo que indicas es una de las cosas que más me enfurecen de las "empresas" (trabajo en IT). Quanta, NETAPP, etc.. que necesidad de borrar todo de internet y de jar a tus clientes (sabemos que el tuyo de 2 hand) tirados? Es que es comos e dice por aquí: "Tirar piedras sobre tu propio tejado"..... Me encanta el canal y todo lo que haces. Sigue así ;)

[@RexorProxer]

Well at least for NetApp the Disk Shelfs there are usable in the Second Hand market, maybe not with the Software provided by NetApp, but the Hardware can still be used.

[@tspawn35]

The enterprise world has no official 2nd hand market. Quanta doesn't delete anything. They have all of it stored in house. They just do not give out any information unless you have a service contract with them. Which once an enterprise server is decommissioned that is the end of the service contract and the servers get sent to recyclers. That's sadly just how it goes.

[@hi-friaudioman]

Nice Klingon Bat'leth on the side of that server! You know you're a TNG nerd when you have Klingon weapons on your server. 😂

[@xavxtx]

Two years ago, I purchased a Supermicro H12SSL motherboard and an AMD EPYC 7282 processor. It has been working flawlessly ever since.

[@darrenorange2982]

You win some you lose some, thanks for sharing your experience!

[@arthuralford]

It's worth seeing how this all ended, and it's a lesson learned that purchasing used server hardware requires being able to have at least minimal support from the manufacturer. It's not like they have to create all the documentation and updates again, or that it takes up large amounts of space on their servers. This reminds me of what happened to Comedy Central and MTV News-years worth of programming deleted because someone thought it was worth saving a few hundred dollars a year over keeping a record that will be useful to someone down the road

[@PaulaXism]

Thanks for the warning. Just saved my other half from wasting yet more money on enterprise grade brickware. He's very good at throwing his money away on discontinued "needs licence but we don't sell it for old junk" rubbish.. I'm running a nice stack of old HPE gear.. nothing after G7.. and I have all the documentation and firmware updates downloaded and safe.. luckily before they voided everything a few months back.

[@TheDwight1379]

Enjoy your HP! Totally agree with your comments on this!

[@the_thunder_god]

Like and a comment for taking one for the team. My efforts recently have been split between success and failure. Success in that nice KVM you did a video on....install went great and I finally have a solid physical interface for my server (which happens to be Dual Xeon E5-2697 v2's). Failure...well I just need to get a different GPU for transcoding for Jellyfin in my server.

[@nrg753]

Ah planned obsolescence. My aunt gave me an iMac from 2011 the other day. I upgraded OSX to the maximum that it can support (officially anyway) 10.13, but it is still very limiting to use unless you want to install really old software on it. So I dual booted Bazzite/Fedora and it's now a great machine! If the manufacturer doesn't want to support something any more, then they should always give the community enough to be able to let them support it!

[@kgonepostl]

Shout out to the Klingon bat'leth on the side of the server, a nice nerdy touch.

[@ws_stelzi79]

I guess there are cheaper and easier ways to get drinking Scotch! 😉 Well, me just bricking a network switch by "just" typing in "some console commands" doesn't sound so dumb and silly! THANK YOU for that realisation! 😇😜

[@MrGpsjim]

Great video. Making a product deliberately obsolete should indeed be a crime!

[@T3hderk87]

I feel your pain Jeff. I just picked up a Thecus W8900. Don't know what that is? Neither does Thecus, or Foxconn for that matter! I had to download drivers for it off of the Saudi Arabian mirror via The Wayback Machine lol, and i don't even have a copy of the LAN driver.... But! I think it is part of the hunt that propels me, and damn is that a sweet server!

[@cameronfrye5514]

Well that was a crappy end to a project! As much as I enjoy seeing the wins, I appreciate when you share the defeats too.

[@seansrightthere]

How we handle failure is equally important as how we achieve success. Thank you for sharing this video.

[@Psikeomega]

The best euphemism for watching the explosion "and now I'm drinking scotch"

[@Prophes0r]

I don't think "euphemism" was the word you were going for. I'm not actually sure what you meant to say....

[@EduardoSantanaSeverino]

Thank you for the video. This server looks really tempting. If any future updates, about this, it would be awesome. And if someone else is able to do the upgrade, it would be appreciated. Probably we can follow this topic on a forum? Or discord? On another note: The first thing I would do with this server would be to quiet it down. And because I only have a short rack, which is NOT that deep, I would put it in the wall. But definitely, this server is intriguing, resource full and promising. I would like to see someone who actually cracked it down and would be able to run epyc Rome in those. Thanks again for the video.

[@r_firefly4292]

Former Quanta Q71L-4U(4 socket E7v2) owner here: Quanta doesn't give a sh*t.

[@SerafKe]

Honestly this tracks with my experience with Quanta equipment as well. Though mine was with one of their OpenFlow switches.

[@chromerims]

You succeeded. In deftly ramping up the drama of this *epic* saga to a high angle. Make the 3rd followup video . . . it will get VIEWS. Desolder, SPI, resolder. Go, go, go, gentle Sir. Thank you for the awesome content 👍 Educational and ENTERTAINING. Kindest regards, friends and neighbours.

[@dragonrider6875]

Hey, I have been there recently and lost a few hundred dollars in equipment. I understand where you are coming from. Love the video's.

[@PaulaXism]

I watch my other half throw money away on this kind of gear. He doesn't do any research first, just "cheap.. shiny... oooooo" and ends up with expensive bricks. Personally I like to KNOW the stuff I buy has zero support but can be dealt with.. anything HPE is no later than G7.. I didn't buy it from them and I don't need to pay them for "support" which they won't provide. It's time we got some laws with teeth for these waste producers.

[@oak409]

Thanks for the video. If you and others didn't post fails. Then the internet would be full of 100% success stories which really sucks when I try and have problems. My depression kicks in an I believe I'm the only one in the world who can't get this to work.

[@coeus1469]

Thanks for sharing this video. You rolled the dice, and came up with a hard learned lesson. Hope someone finds a method for you to get to the IPMI eventually.

[@Byahn-Provelocity]

3:20 Is that a legit metal Bat'Leth? or is it lightweight sheet aluminum. :D Is it sharp?

[@CraftComputing]

It's an official recreation from the 90s. Screen accurate and made of aluminum. Not sharp at all.

[@Byahn-Provelocity]

@@CraftComputing Man, I so want one that's made of 440a steel, and sharp... to hang on my wall by the front door of my house. Crazy peeps come 'round, they ain't comin in.

[@mattliston1215]

up front, honest, and not rudely throwing someone under the bus. Just being open and blunt. Not a single problem with that. Keep up the good work! I also do not understand some company's need to add e-waste via lack of software documentation. Shoot, you can still download drivers for some computers made in 2001 on some brands websites!!!

[@PaulaXism]

Packard Bell (Asia) had drivers and bios for a Pentium 2 last week when I needed them, free for download !!.. Now that's impressive service.

[@Freedbot]

Respect for attempting the direct chip flash, and more respect for knowing when to call it quits and posting the video anyway. It's good for people to know how these projects can turn out and to be ready for it.

[@SP-ny1fk]

I wish enterprise manufacturers were more engaged with the community. I would like to see the emergence of a joint archival initiative for older systems - BIOS, documentation, old discussion threads....

[@njkoch]

"And now I'm drinking Scotch" sounds like me every time dealing with these companies.

[@bobsodman5274]

You had me at "I think we're gonna YOLO it"

[@thetj8243]

As soon as he pulls out the Scotch bottle, you can be sure this is going to be a post-mortem video. I think this kind of video is important to educate about bad business behavior and possible traps you can encounter using used enterprise hardware

[@ciaduck]

I had some issues doing In System Programming of an 8 pin BIOS chip in a laptop. Eventually I figured out that my programmer (raspberry pi) was not able to supply enough voltage to the platform to program the chip. I disconnected VCC and ground on my clip, plugged in the system, and allowed it to enter an "on" power state to drive the chip for programming. I don't know if you can safely pull that off in this scenario, but it would save you some nasty soldering job. Love the videos. I really enjoy your "cloud gaming" series.

[@VintageSupportServices]

If you have original bios backup's, I'm happy to do a chip off bios flash for you, Much less complicated than the skinny sprint v2 I designed for epictronics!

[@lukes1978]

Maybe the board has an IPMI/BMC override jumper? When you set that it accepts any password. I have seen that on a few older systems.

[@gedgicat2063]

Feel for you, been in a similar place before, I was lucky and was able to buy dual bios chip add-on with a selected bios flashed on it. It worked for me and saved my bacon a couple of times down the road. Having a hardware locked bios chip is a pain. Only option I've seen in cases like this is to find pin in question and disable it or as you say remove the chip. Don't know if its possible to fit a chip holder to make life easier going forward. Best of luck off screen if you care to revisit to prevent the inevitable E-waste👍

[@SidebandSamurai]

Jeff, I am sorry this happened to your server. I love your channel, and failures are just as great to watch as your successes. Is there a JTAG port you can push a BIOS update to instead of removing the chip. Hey look at the bright side. It would make a really good video of you doing surgery to the board and removing / reflashing the bios chip. I would watch. I am quite sure you have thought about this but thought I would put my 2 cents in.

[@WXSTANG]

I learned when I bricked a video card that if you go to the manufacturer datasheet for the bios chip, there are sometimes recovery methods to roll back to the previous bios. I ended up jumping two pins of the bios and it rolled back to a previous bios. Might be worth a try.

[@truckerallikatuk]

Your first video on this machine put Quanta on my "Do not buy" list, this one confirmed it.

[@titaniummechanism3214]

The fact that Quanta was so unhelpful when you reached out, probably soured tens on thousands of your fine viewers towards their brand even more than they were already. Not a good move.

[@Trylen]

Wow, ouch. You have my deepest sympathies. I was hopeful you'd get this running. This weekend, I might open my bottle of mead to drink to your pain. Die Hörner Hoch (Raise your Horn) Hey if it makes you feel better, I grabbed copies to archive and while prepping the downloads, I'm going, "the the hell are my other drivers?" While I watch youtube, it's off a remoted PC so I can pick up from where I left off anywhere in the house... which means off my main PC, it sees different drives....IQ143 yup I a genius... they need to raise that bar...only took 10 minutes to figure it out...

[@DavidHunterChunkyRiffs]

It happens to us all and is part of the learning process we all love and hate :D I'm grateful you posted this though, it's important to show the ups and downs.

[@thirdwheel1985au]

Watching this wondering when the scotch would come out... And then ahhh there it is

[@rfitzgerald2004]

You mentioned it was a dual-node system. Does the second node work and could that be used to rescue anything or could you clone the good bios from the second node to the first with an EEPROM programmer?

[@CraftComputing]

That's basically the only hope I have left.

[@BrunodeSouzaLino]

12:54 This is so in line with how modern moderation on some sites work nowadays: "You've been banned from violating our terms of service/rules, but we're not gonna tell you which rules you broke."

[@codeman99-dev]

Are you going to jump on the "Let's build a 10-inch rack" train? Raid Owl just did a build. TechnoTim is up next. I would love to see your version. 😃

[@CraftComputing]

Hahaha, definitely have been pricing out my dream 10-inch rack :-)

[@marcogenovesi8570]

his rack is more than 10 inches

[@jttech44]

@@CraftComputing Counter point, 12.25in rack, so you can use the standard ikea kallax cube shelves.

[@alexlovett1991]

@@CraftComputing+1 for your take on the mini rack!

[@nuvotion-live]

You’re giving up such a big ecosystem by making that jump. And is there even any 10 inch UPS in existence?

[@rkessing01]

Forced obsolescence is the name of the game in the electronics industry. I've seen this quite often over the years in embedded electronics, especially with FPGAs. Especially frustrating is that the next generation replacement parts are not pin compatible, not even in the same package. So, you have to redesign your circuit board in addition to redoing the FPGA design itself. And the design software needs to be updated to support the new parts. That requires a yearly subscription to even come close to being economically feasible. They get you in multiple ways.

[@bruhmoment-42069]

Looks like i am RMAing all the quanta stuff my org bought in the last couple months. $750k of server hardware with the expectation to be able to use it for 6 years. It will be easy to justify this to leadership based on the risk of drivers disappearing at any time in the near future.

[@BastienAuxer]

You can still get a video or two out of this. Showing us how to desoder, flash, and soder back on. Just not now. Scotch time.

[@azandy99]

It's always good to include the failures as well as the successes. All too often, there are failures more than success.

[@markarmstrong2746]

From my experience working with enterprise hardware that has no support for the application I wanted to use it for, the whole point of getting the bios to work is to try again. If one bios does not work, try the other one. Also, I would suggest adding a socket to the board and a header to the bios chip for ease of testing. Modding can be done with the bios, as long as understand how it works.

[@AC3handle]

I'm feeling shades of Louis Rossmann here.

[@Ryet9]

What are those CUBES? Sorry for your loss, Enjoy the Scotch! Edit: Explained at the end - Whiskey Stones; linked in description on craft computing site

[@DrywFiltiarn]

I'd think the BIOS on these boards is like nearly any other motherboard nothing more than a simple EEPROM chip located somewhere on it. With that it means you can simply grab something like a CH341A programmer to flash the EEPROM to whatever. As to what you are saying about the voltage drop when you hook it to the board, that's a common thing happening on various x99 boards from China as well, where we're not unfamiliar with flashing the EEPROM's directly. In order to resolve it in those cases, you will need to have the PSU connected and powered (without actually powering the machine itself). The permanent powered utility lines of the PSU will rectify that voltage drop.

[@Prophes0r]

He tried in the first video. He would have to desolder the chip and flash it externally, which is either beyond his skill, or not worth the effort.

[@kuhrd]

I had a similar runaround with HP for a server that was EoL but still in service by the place I worked art and thousands of companies who were eventually going to upgrade but the servers were still doing what they needed to without issue and could get a new lease on life with a simple BIOS update to support the next generation of processor but they were gatekeeping a BIOS update that existed simply because they wanted these customers to buy new hardware even though they were still paying for a service contract. It ended with replacing the HP servers with Supermicro units.

[@sstreet9436]

Sing with me: "He had HI hopes, he had HI hopes..." Sorry Jeff, as you said, this is all too common in the computing/electronics market. Phase a device out and forget it existed. Somehow, I guess I've gotten lucky; I've got several HPE servers that EOL in 2017,2018, and 2020 and thankfully HP still has all the most recent Bios, drivers, and QVLs available on the website - buried mind you - but there. I'm pouring a glass of single malt in honor of the frustration that is our hobby. Cheers!