FIX: Embedded Flash/SD-CARD Degraded issue on HPE Proliant servers

Formatting the embedded NAND flash of HPE Proliant servers is recommended in article c04996097. This procedure is recommended if you see in the iLO GUI the follwing errors: "Embedded Flash/SD-CARD Degraded", embedded media manager failed initialization, the iLO works slowly, or you are unable to collect AHS log.
Advisory:
support.hpe.co...
Intelligent Provisioning for Gen9 Servers
support.hpe.co...
Intelligent Provisioning for Gen8 Servers:
support.hpe.co...
HP Lights-Out Configuration Utility:
support.hpe.co...

[@dudindmitri]

HPE knows very well that after you got an ILO Health degraded status due to the corrupt or worn NAND flash memory there is not much you can do to solve it. I work for support company, and world wide we replace so many Gen8 system-boards on a daily basis. In addition same issue replicates in Gen9... Its a great method to force customers to get a new hardware because issues usually starts after your HPE official warranty is expired. They could just allow users to use internal USB slot for alternative NAND flash memory source when onboard memory is worn out, but no way.... money money money.... what about reputation?

[@Erikih]

Totally right imho. I have a DL360 G8 server over here and i tried to format the NAND Flash memory, but still a degraded status. But it actually doesn't impact server performance. The only downside of this situation is that you can't use Intelligent Privisioning anymore, and other handy tools. But in case you want to add a new RAID array, you can use the Option Rom for arrays utility. By pressing F8 on POST.

[@dudindmitri]

@@Erikih Hi Erik. If you are a private owner of the the server then you can survive without ILO functionality, Intelligent provisioning. *.ahs logs and etc. For RAID you can also boot from Smart Start CD/USB and run ACU. We provide support contracts after OEM end-of-service dates and I don't remember a case where customer accept to disregard ILO Health - Degraded status. IN addition, several of those customers do not allow any FW updates... I had cases where system abnormally power cycle reporting only NAND related issues, sometimes ILO become slow, some customers reported that FANs are working on Max speed and its too noisy, all caused by NAND flash memory issues. Whats interesting is that I did format NAND from ILO for our LAB server and for 2 different customers. For both customers ILO reported degraded status after 3 months and same happened with our LAB DL380 G9. After that I tried to do NAND format for 50-100 HP Gen8 cases and for 80% cases ILO is degraded again after reboot or within 5-10 min, and for remaining 20% it will get degraded again within hours. So we have to replace SBs all the time...

[@toerti9589]

@@dudindmitri you could also mount/open the Intelligent Provisioning Recovery image, in the folder ip are several other images, extract the gaius.img.gz, rename the extacted file to gaius.img (should be about 1GB), mount that image to floppy and boot the server. Then grub boots and you should be able to select Inteligent Provisioning or Smart Storage Administrator. Super annoying but at least it lets you manage the Raid Controller when the NAND is broken.

[@dudindmitri]

@@toerti9589 Nice. Never tried this method.)) I usually use smart start bootable media and access ACU from there.

[@toerti9589]

​@@dudindmitri yeah, i never found that on the hp website, hence this odd method. They don't realize that people are not going to buy another hp device if the support is that bad for the old machines.

[@dudindmitri]

In some cases even after formatting NAND, ILO will become Degraded again shortly. In that case you can try below: 1. Full Power Cycle may help (remove all cabling and wait for several minutes) 2.Update Intelligent Provisioning Recovery Media 1.71 for G8 and 2.71 for G9 (Mount iso in ILO GUI, One Time boot from it and update) 3.Some people reported that first you need to reset ILO to Defaults then format and it will stay Healthy. Strange thing noticed: I have fixed 2 cases with 2 different customers by reinstalling HPIP (Intelligent Provisioning Recovery Media) and same time solved same issue on our lab server DL380 G8. So in 3 mons +- 10 day both customers reported that ILO degraded again and our lab server also got ILO health degraded even its not under any load....

[@alexan_]

добавлю к видео, что ошибка пропадает только после полного обесточивания сервера (обычный ресет ILO или ресет сервера результата не дал), я делал многократно формат с версии 1.71, после ресет, заходил обратно в диагностикс - там все равно висело "деградед", прошивка провизионинг с iso образа заканчивалась ошибкой flash error, а потом обесточил сервер, зашел в ILO а там health ILO ok.

[@Gladiuss]

Мне помогло обесточивание сервера и повторный формат флеш.

[@alexan_]

@@Gladiuss ну или так, или формат проходит в предыдущий раз, но Ilo не обновляет статус. Все верно .

[@s541263]

Блядь ну почему вот такого простого шага нет ни в одной инструкции???!!! Целый день убил на это говно! Выключил из розетки, включил и все получилось! Спасибо тебе добрый человек!

[@Da_808]

+100500 в карму, форматнул из веб, форматнул как тут командной со скриптом, уже на все плюнул но случайно наткнулся на пост)))

[@user-dn7xe5gt8o]

Если не помогло, пишите решу эту проблему)))

[@eyurtese]

The video is using a convenienty not broken machine to fix it. I updated iLO and then saw this problem. But it is not possible to fix with suggested methods.

[@samanrad]

Hello. I have the same issue. Have you fixed that?

[@isaiahwinbrone]

What about technology in the year 2020

[@abc-if3rs]

very helpfull thx

[@gabrielbedell308]

If i perform the nand format...Will it not affect the OS that is currently on the server?

[@joele.3600]

no, the NAND is an internal storage area that iLO uses to store diagnostic data, and does not affect the installed operating system. You will need to reinstall Intelligent Provisioning, however.

[@aaaaaa_1977]

Кто то знает где на плате эта NAND Flash располагается? Уже с десяток gen8/gen9 скопилось таких, и формат и бубны не помогают никак.

[@dpotseluev]

Возле задней стенки, либо Hynix либо Sandisk SDIN7DP2-4G.

[@professor_trottelrainer]

Нет, он находится на обратной стороне платы, под слотом для SD карты. это Sandisk SDIN7DP2-4G. Теоретически, его можно заменить. как я понял, сейчас - 4G уже мало где найти, есть 8g, они большего обьема. У меня есть такой сервер, хочу попробовать заменить этот чип.

[@professor_trottelrainer]

@@dpotseluev вы пробовали его менять ?

[@aaaaaa_1977]

@@professor_trottelrainer то то его сверху нигде не видно! поэкспериментирую у китайцев по $5 есть, придется подождать

[@professor_trottelrainer]

@@aaaaaa_1977 я создал на Ixbt тему в серверах, там все собрал, и там есть ссылка на тред с фото. меняется он гиморно, нужен реболлинг и навыки. сейчас чипов на 4г, как я понял, нет, в основном большего обьема, теоретически должны подойти

[@flaviohernandez4796]

This process works for windows servers, do you know how to do it for VMware Servers?

[@ZZQ95]

在查阅大量文档、资料和网站后，我得到了一个可怕结论，HP Gen8、Gen9上的NAND芯片事实上是个设计缺陷，它极其容易磨损，对NADA格式化操作不会好用多久，很快就会坏

[@TonyMontana127001]

Музыка как в "магазине на диване"

[@ozgurXeryigit]

Allah razı olsun kardaş

[@aliamani4502]

does it need to be done ? or the ilo will keep working ? i mean if i don't wanna format or replace the system board .

[@user-vs3sh5eb4n]

iLO will continue working, the server will continue working. But if something happens with the server you will not be able the generate an AHS log and HPE wont be able to diagnose the issue.

[@dudindmitri]

@@user-vs3sh5eb4n AHS log is useless if your HPE support contract is expired. Its possible to troubleshoot and find the problem without ILO or AHS. Boot POST messages and/or system minimum configuration will give you the answer.

[@mohamedchabanne7191]

hi i need ,help my server hp dl380 intelligent proviosinig F10 is missing on startup ,and thanks

[@samanrad]

Hello. I have the same issue. Have you fixed that?

[@user-dn7xe5gt8o]

Кому не помогло , решите решу проблему) Территориально нахожусь в мск)))

[@professor_trottelrainer]

Прошу написать, есть такая проблема на 3 серверах.

[@MaximAhtyamov]

Как решить эту проблему?