Dell PowerEdge R820 R720 storage upgrade Part 3 | How to upgrade with NVMe U.2 SSDs!

[@ThomasTomchak]

As much as I learn from the main topic of any one of your videos, I learn equally as much (or sometimes more) just watching how you work through problems, what your thought process is and what tools you use. In this video, I love that you kept in the mistakes or unexpected results and reasoned out some of the possible reasons. And I can honestly say there is no other channel I follow that I look forward to having long videos like yours. Normally if it's over 10 minutes I'm debating to even watch it, but for yours, I go refresh my coffee and look forward to watching it start to finish. Even though this is not something I plan to do with my R720, I really enjoyed the whole video and learned a lot about your process. Thanks as always for creating such a great and informative video.

[@ArtofServer]

Thank you Tom for your kind words! :-) You may be one of the few, and I very much appreciate that you watched the whole thing from beginning to end! Also, thanks for tips on mics. Hopefully, you've noticed an improvement in audio quality?

[@ThomasTomchak]

@@ArtofServer OMG yes, the audio. How could I forget. I noticed immediately and smiled. Thanks for the upgrade. It sounds great now.

[@oksofast]

i am testing a Fujitsu D3262-A12 Controller U.2 NVMe SSD card PEX8732 in this configuration, hope this work for get pcie3

[@ArtofServer]

Sounds cool! Good luck! :-)

[@stolenmynametyyoutube]

So i have watched, rewound, many times but some how managed to got it done, but i have also discovered something in the process... So seriously not a joke I bought as a joke a nvme single slot pcie card for $5.96(dollars) needed to spend atleast 10 dollars on TEMU so i got headphones lmao, but i programmed one following this tutorial and plugged the cheap one in and I didn't have to set it up infact I did nothing but it showed up. So 5.96 keep that in mind.

[@ArtofServer]

Thanks for watching the video! :-) 😁 Those m.2 cards are great. But only 1 SSD. This setup was for U.2 and for 4 SSDs. Quite a bit different. The quad m.2 adapters require pcie port bifurcation which is not available in this generation.

[@peterdee1900]

Your linux sys admin skills of using lspci are outstanding!

[@ArtofServer]

Thanks! :-)

[@rinku-blogadmin]

Sir , what if you have twisted the cable , may be you do not need to cut the tape . Simply swith all ends to hard disk side from RAID controller

[@ArtofServer]

I'm not sure what you mean? The cable I cut was for the NVMe connection, not for any RAID controller?

[@rinku-blogadmin]

@@ArtofServer there is two end of the cable , one side there was masking tape otherside no masking tape , i was just saying if you have taken the masking tape side to other side you no need to cut masking tape sorry for writing raid controller

[@HyRax_Aus]

Great video - love to see what simple mistakes can be made and how they can be avoided. As someone else said, it's also somewhat therapeutic to watch server builds in general too! I upgraded my R720 from 8 to 16 drives using this video as a partial guide when I didn't notice the screws hidden under the SAS cables and couldn't remove the blanking bay, and found it was definitely easiest to install bare cage, then the 16x backplane and then drives as well.

[@ArtofServer]

Glad you enjoyed it. thank you for watching!

[@drcemdede]

Thanks for the video ! May I ask about the extension board that helps doubling the CPU number, does the R830 also has this capability?

[@ArtofServer]

I don't have experience with the R830, so I'll have to make some assumptions. But I think the R820/R830 series is Dell's 4-socket in 2U form factor servers. If that assumption holds true, I expect the R830 is similar, but with newer technology based on the Haswell/Broadwell generation.

[@drcemdede]

@@ArtofServer Thank you very much for the reply.

[@HotloadsTTV]

Great series on the R820! As a proud new owner I learned a lot! You mentioned iops and I would be interested in a video that explains all the elements that drive and system that impact iops.

[@ArtofServer]

Glad you enjoyed it! I think a good place to start is to watch this video: kzbin.info/www/bejne/bIrHhp-EZ7uNack

[@woxit6107]

@@ArtofServer Nice suggestion, thanks.

[@mranthony1886]

Shame you didnt do a single Nvme in the slot running at full speed

[@ArtofServer]

I did at 41:09, no?

[@Ryan-zer000]

1:03:41 thanks, that's exactly what I needed to hear for my setup. I was debating on going NVME (saw some decent related deals on eBay recently), but it really seems that this NVME solution for the R720 and R820 doesn't provide much benefit, especially with the added costs in server components and drives.

[@ArtofServer]

I think the parts are cheap enough these days, there is still some benefit to this particular setup. You still benefit from the much higher IOPS capability of the NVMe drives, just a limit on the throughput. There's a newer version of this kit from the 13th gen with a PCIe 3.0 switch instead, and sometime down the road I want to try swapping that part to see how it improves the throughput.

[@Maxx-qq8lh]

@@ArtofServer - many thanks for the Info. Could you pls. supply us with the Dell Part number of that Dell 13G Kit (with the PCI 3.0 switch) ? there's a real chance for these to work in Gen12 Servers also. I also found out that while Dell 13G servers initially did NOT support PCIe Bifurcation, support for PCIe Bifurcation was added in upcoming later BIOS revisions. Still would be interesting to test a Dell 13G Kit WITH a PCIe Switch - which oviously does NOT require Bifurcation.

[@UnkyjoesPlayhouse]

another great video :)

[@ArtofServer]

Thanks man!!!! :-)

[@woxit6107]

Thanks for your time and effort.

[@ArtofServer]

Thanks for watching! What's next for you? ;-P

[@woxit6107]

@@ArtofServer I've set my eye on an other playlist, I'll see you there. ;-)

[@propeto13]

Is there an upgrade path for R730XD LFF for either the rear 2 drives or any of the drives in the front for NVMe U.2 SSDs drives? any recommendations be greatly appreciated.

[@ArtofServer]

Not for the "XD" models that I know of. What you can do instead is use a PCIe M.2 adapter. This generation supports PCIe port bifurcation, so you can even use one of those quad M.2 adapters to put 4 NVMe drives in a x16 PCIe slot. But I don't know of any U.2 solution except for the SFF models which can take a special U.2 backplane retrofit.

[@Tindog81476]

I did this upgrade on a Dell r820 and I can confirm that this does in fact work on Unraid, and the pcie card works with Unraid also.

[@ArtofServer]

Cool! Glad to hear it!

[@aperson9495]

I don't know if they made or were thinking of making a R820xd, but the three Backplane power and data connectors would probably be needed for the 24 bay R720xd, which I think consists of three 8-bay modules.

[@ArtofServer]

The 24-bay backplane in the R720XD is a single 24-bay backplane, not 3x8, but you've brought up a good point.. even though it is a single backplane, it has 3 power connectors to power the 24 bays! So, yeah, now that makes more sense for the extra backplane power... but not the signal port (only 1 signal port is needed for the 24-bay backplane). That said, I don't think there was ever a R820XD with 24-bays. But, I can also see them just re-using the motherboard design between the R820 and R720. thanks for watching!

[@TheInternalNet]

Wonder even with a 16 bay if it can be converted to a 8 bay PCIe NVMe. Drastically change the performance of the whole box. Amazing channel and video. Been drooling over a 720 16 bay to upgrade my r420 4 bay.

[@ArtofServer]

Thanks! Yeah, I just wish the PCIe switch card was PCIe 3.0 instead...

[@TheInternalNet]

Is there a 3.0 version on the market?? You also said in the video that your video that the drives you were using were causing issues with the test. Would you work with similar setups and give how tos??

[@ArtofServer]

@@TheInternalNet there is, but it uses different connectors. I haven had the time to investigate further. Yeah, I would use faster u.2 SSDs if I had the PCIe 3.0 switch.

[@mishanwarnakulasuriya1346]

@artofserver whats the best way to add a host drive that is separate from the H710 on a Dell R720? I need to pass the H710 to TrueNAS and be able to run Proxmox on a different drive thats not on the H710. thank you

[@ArtofServer]

You have several options, depending on your R720. For one, there's a USB-A port on the motherboard that you can use an adapter to run a small SSD. (I demonstrated this in R410 series) You can use that USB-A port in conjunction with a PCIe NVMe drive but boot strapping off the USB drive but have the rest of the OS and your VD for TrueNAS on the NVMe PCIe drive. (i've demonstrated this on R510) You can even have 2 such drives to mirror them. If your R720 has an optical drive, you can buy an adapter to replace the optical drive with a 2.5" SSD. (i've demonstrated this with R710 series). If your R720 is a "XD" variant, you could install the rear flexbay kit and split that off from the front backplane. I'm sure there are many more options I can't think of right now.

[@mishanwarnakulasuriya1346]

@@ArtofServer Thank you for the quick reply. Its DELL R720 8SFF Model, not sure if that supports the flexbay, but I think that would not solve for the issue with the H710 needing to be bypassed from the host drive. Unless there is a way to get that kit and connect it to a separate raid controller? I'm definitely interested in the optical drive option, hopefully that does not have an impact on performance.

[@Maxx-qq8lh]

@Art_of_Server - first of all big thanks on all the material you share with us, truly a fountain of information. I'm somewhat familiar with NVMe/U.2 storage - at least that's what I believe, so let me share my 2 cents. First of all, many server boards of that generation e.g. Xeon E5-26xx or E5-46xx experience an odd behaviour with SANDY Bridge CPU versions - often limiting PCIe bandwidth to Gen 2 speeds (check out many Intel server boards discussion threads e.g. SC2600P series), so one has to use IVY BRIDGE Xeon E5 chips to ensure Gen.3 PCIe can be obtained. Now on connecting so many U.2 (which is merely an NVMe implementaion just as M.2) SSDs (4 in your case) you must know that each connection wants to make use of 4 PCIe lanes each (if unavailable, the connections revert back to 2 PCIe lanes), requiring 16 PCIe lanes. Later concepts (e.g. Supermicro X9D... or especially X10D... mainboards) allow PCIe BIFURCATION. Not sure whether DELL supports that on that Gen 12 (?) platform you are highlighting here (R820, R720, R520, T620 etc) but for their NVMe upgrade kit they chose another route - the PLX Switch, which merely breaks up a 16-lane PCI slot into 4 PCIe lanes each to drive in total 4 such NVMe SSDs. Note that in theory, other PCIe Switches from different vendors might be employed in lieu of that slow PLX switch. Have you or anyone else tried to merely replace the card with the PLX switch ? There should are cards available from third parties (Supermicro might be my best bet here). However not sure how such "foreign" cards affect the iDRAC system management behaviour engineered by Dell. I've seen nightmares (e.g. Lenovo's recent servers) where all fans suddenly run at full speed once you add just one single piece of "foreign" hardware, even HDD or SSD. Any insight appreciated ...

[@Maxx-qq8lh]

Update: just checked Amazon, found this card (chinese OEM) that could be interesting here ASIN B08F2DK6G9, but right now unavailable.. according to the doc, this one has a PCIe 3.x switch onboard. The cabling would need to be changed, though

[@ArtofServer]

Yes, I've experienced the PCIe 2/3 issues on Supermicro X9SCM-F boards too! However, I don't believe that is an issue here with Dell's 12th gen servers. The issue with PCIe 2.0 is simply the older PLX switch they used in this particular "kit". Yes, PCIe bifurcation in some ways negates the needs for PLX switches in use cases like these. PLX are still useful if the platform simply doesn't have enough PCIe lanes, like systems with with large array of U.2 drives. AMD Epyc is making things easier now with the abundance of PCIe lanes though... Thanks for watching and commenting! Hope you find the rest of the channel useful! :-)

[@Maxx-qq8lh]

@@ArtofServer, thank you for your cogent reply. You are 100% "spot on" with your comments about the beauty of AMD's Epyc - with these many lanes and the continual need for PLX switches - after all, you max out with 4x NVMe connections (4 PCIe lanes each) with PCIe-Bifurcation methods.. PLX Switching easily bypasses this bottleneck.

[@Maxx-qq8lh]

@@ArtofServer - FWIW, I found almost all of your posts and playlists useful for my own endeavours.

[@LuminousPlasma]

So can you boot from this, even though the 12th gen servers don’t support bifurcation?

[@ArtofServer]

No, I don't think so. You would have to boot off SAS/SATA/USB device and use this just for data storage.

[@liamgamache]

Is there a way to get another 4 NVME drives in that caddy?

[@ArtofServer]

Not with an official kit. there might be a way to retrofit something but I haven't tried.

[@liamgamache]

@@ArtofServer good to know. I have 8 P3700s i got for a steal I want to use for my truenas server.

[@Maxx-qq8lh]

Maybe one has to buy another of those "caddyies" for the official kit and populate the other part of the caddy, now "sealed" with a blank-inlet. According to your video, these are separable. Note that there exist 8-port U.2/NVMe cards - even 12-port cards which employ PLX (or other - Marvell and ASM come into my mind) switch chip from Broadcom. Although I only found affordable (sub $100) sources for a 4port generic (China-made) PCIe 3.0 HBA with a PLX chip.

[@Maxx-qq8lh]

Just realized to achieve this, the backplane would need to be modified also. I will look into this over the next couple of months myself as I just bought a Dell Poweredge R720 in Germany which I will pick up later this year

[@davkenrem]

Did the card require PCEI Bifurcation?

[@ArtofServer]

No, PCIe port bifurcation isn't supported in this generation anyway. And the point of the card is that it serves as a PCIe switch so you don't need bifurcation. Good question though.

[@baudneo]

Awesome! Always enjoy your content!

[@ArtofServer]

Glad you enjoy it!

[@VP-xz8hz]

Awesome video!

[@ArtofServer]

Thanks for watching!

[@BillWood1]

This series is great. However, is there an equivalent expander kit (or maybe this one?) that works on the R730? Not the XD version, but the regular R730?

[@ArtofServer]

Yes, I believe there is. I don't have a R730 so I haven't tried it myself.

[@n.aminr.7175]

Does this apply to R730 as well?

[@ArtofServer]

The R730 does have a similar option but with different parts. I think they use a PCIe 3.0 PLX switch.

[@n.aminr.7175]

@@ArtofServer good tip. Thanks

[@bentheguru4986]

Sorry, skipped a lot of the video as time is short. No PCIE Bifurication chganges in BIOS seen. Was surprised to see the 820 is only 2P as the 800 series were 4P designs. Anyway, for all this work, a PCIE adapter card (cheap and very available on eBay) 4x NVME drives attached, turn on PCIE bifurication and now you have 4 NVME drives that are also very cheap to use. Front bays can stay as SATA/SAS.

[@ArtofServer]

not sure what you mean by 2P vs 4P? If you mean the number of sockets, the R820 is a 4-socket system... they are stacked one on top of the other. see other videos in the R820 series. yeah, PCIe bifurcation is not supported from what I know. so a quad M.2 NVMe is not an option here unless you buy one with a PCIe switch on it. thanks for watching and sharing your thoughts.

[@bentheguru4986]

@@ArtofServer Ahhh, never had a 820 on my bench, 4P it is. PCIE Bifurication, appears that it only came in on the x30 series. Not the x20's. I have it here in my R7910 WS that is basically a GPU friendly version of a R730. Keep up the videos, good value, even when they can be a bit long. Would love to see a Rack or workshop tour.

[@ArtofServer]

Thanks for watching. :-)

[@dyong888]

Great video. With the YPNRC PCIe switch expansion card 4-port, is there an equivalent for Supermicro X10 generation motherboards? I'm running an X10SLH-F and X10-SRM-TF motherboards. I may also be adding a X10SRH-CLN4F in the future too. The last two boards use E5-2600v4 series xeons and are great for workstation / virtulisation use.

[@ArtofServer]

I love the X10SRH-CLN4F! Actually going to review that board in the future on this channel. I don't know of a general PCIe switch card that can be used on the Supermicro though... Dell has one from the 13th gen that is PCIe 3.0... could be interesting to try it out.

[@dyong888]

@@ArtofServer Yes. It'll good to see if any pci-e switch card works on Supermicro motherboards or if Supermicro has their own. On a similar matter, do you know any pci-e card that lets me mount one M.2 NVME SSD? Only one so as to avoid pci-e burification which the Supermicro motherboards do not have. I tried two brands on the X10-SRM-TF and only the Asus Hyper M.2 X4 works. Its been discontinued though. I got it for $12 dollars years back and only tried it recently. Am planning to install one into a future X10SRH-CLN4F or on my other older mobo the X10SLH-F as these motherboards do not have on-board support for m.2 NVME storage. I also found out that the X10-SRM-TF which does take on-board M.2 NVME will not see the Samsung 980 Pro. I have a Samsung 950 serving as boot drive all these years which works. I suspect its because the 980 pro is PCI-E 4.0 even though they should be backwards compatible. Hence my search for PCI-E expansion cards to mount an extra M.2 NVME drive in my motherboards. What are your thoughts?

[@ICanDoThatToo2]

Regarding the connector -- that's actually a U.2 connector, and not necessarily NVMe. U.2 is intended to support NVMe as well as SAS and SATA, so the mere existence of that connector doesn't tell you which interface it uses. I have an HGST Ultrastar SS200 that has all the pins on both sides, but it's just SAS 12Gb.

[@ArtofServer]

No, I believe you are mistaken. U.2 and SFF-8482 may look similar, but they are not and cannot support all those protocols. There's a video on my channel showing how they differ physically.

[@alexanderhunt9263]

​@@ArtofServer U.2 and U.3 carry SATA lanes, however you're very limited on what the motherboard supports.

[@JudgeDrD]

@Art of server does this card activate the storage and chassis sensors in idrac? We're using hba330, with no amber light issues at all, but we don't get these extra sensors activated in idrac.

[@ArtofServer]

Not sure what you mean?

[@JudgeDrD]

@@ArtofServer the HBA you suggest using - does it activate the storage section in idrac?

[@caseyhouston1921]

Is this possible on a R740?

[@ArtofServer]

I think R740 already has provisions for U.2, no?

[@Maxx-qq8lh]

@@ArtofServer , to my knowledge (from the manual) only the R740xd seems to allow U.2 drives

[@jj-icejoe6642]

So nice thank you !

[@ArtofServer]

Thank you for watching!

[@itsbackwards7773]

I ordered the parts from a different list that didn't include the backplane data cable. I want to mention that everything works fine without it but you do get a boot message requiring pressing F1 to continue booting. I ordered a cable so I can skip F1.

[@ArtofServer]

Do you know if the slot online/fault status works correctly without the backplane signal cable?

[@oksofast]

cant install 2 nvme kit on same server and boot from sata-dom??

[@ArtofServer]

In this R820? you would have to figure out how to power the SATA-DOM. It probably can be done. I don't know about install 2 of these NVMe kits though..

[@o_poida]

Good day, Art of Server! Can I ask You two questions about sas expander Dell YPNRC. 1. The cable from the store Aliexpress SFF-8087 to 4*Sata fits to it? (Mode HBA Sata 16*HDD) 2. How to find out the operating mode output for x16 to x8x4x4 or output for x16 to x4x4x4x4 if taken separately without a kit Dell? There is no such information anywhere at all! Thanks in advance!

[@ArtofServer]

1. I don't know what cable you are specifically referring to. 2. In linux, the command 'lspci' will tell you how many PCIe lanes are connected for each device under "link width". If you connect a U.2 drive to the YPNRC, lspci will tell you how many PCIe lanes are connected to the U.2 drive. I think I show similar commands in this video so check it out.

[@andriitarykin9567]

Thank you!

[@ArtofServer]

Thanks for watching!

[@elcolin_]

Those aren't Intel Optane drives, right? That'd be so cool if they were. I wish you'd make a video about anything Intel Optane related! I just recently got into them and went down the rabbit hole. Total shame Intel killed off Optane. But I would LOVE to see more videos of servers rocking some Optanes either as a boot drive or in some storage configuration maybe RAID. There aren't many videos out there so you could break into that niche. Idk, I'm just an Optane fanboy. I recently figured out how to get my PowerEdge R430 to boot off an NVMe M.2 SSD through an AIC on one of my two available PCIe slots. So yes, it can be done, even though people will tell you it cannot. I'm tempted to put this Optane P1600X I just got as use it as the boot drive for this R430 server. It's crazy low on capacity though, especially for the price :( 118GB is basically just for the boot drive only. I guess you pay for the crazy cool 3D Point technology, random speeds & low latency as opposed to capacity.

[@ArtofServer]

Yeah, Optane is now obsolete in the server world so I guess they are flooding the market. I'll put it on my list of future videos. Thanks for the suggestion and watching! :-)

[@drtweak87]

Looking at the specs for the Card, it did say it offers 32 PCI Express Gen 2 lanes. Maybe the reason why it runs at Gen 2 is so that when you plug it into 16x Gen 3 PCIe you can run 32 Lanes at gen two? half the bandwidth but double the lanes?

[@ArtofServer]

No, I don't think that's how the lanes are distributed. It's x16 + x4 + x4 + x4 + x4 = x32. The x16 is on the riser, while each of the x4 go to each SSD. So, all x32 lanes are being used at PCIe 2.0.

[@dusterl1472]

You know, I *was* going to get an R710 to replace my R410, but here you go all showing off what this generation can do... G'damnit. Though Im curious... I assume the pcie card uses a pcie switch (everything I can find says the 12th gen can't do bifurcation), so if one is running a hypervisor (like, say, ESXi or unraid) can these drives be passed through individually to VMs?

[@ArtofServer]

Ha ha ha... an R820 or R720 would be a very nice step up from R410. I think R410 is great starter, but there does come a point where an upgrade will be needed. :-) You know, that's a very interesting thought about the PCIe passthrough of the SSDs to the VM. I suspect it will be possible, but I haven't tried it to confirm. Perhaps a future video idea! Thanks for watching! :-)

[@TheInternalNet]

I was thinking the same thing for my T3500 tower. Run Linux Raid on it which Wendell just did a video on. Would absolutely scream. Might be worth doing the R&D

[@also5015]

nice video! question. how are the SAS cable for 4x SAS work for nvme? also do you know if the R730XD mid tray cage riser work in the 720? btw if its R720 and sandybridge xeons, they are PCIE 2.0 as well so even if the card is 3.0 it will not be 3.0, also the overhead for 2.0 encoding is 8/10b so 2000 MB/s is actually around 1600MB/s ish max, and even if that is the case the PLX chip on card might have it's own spec and likely much lower than what 2.0 PCIe x16 lanes can do.

[@ArtofServer]

SAS cables used for NVMe are not transmitting SAS protocol. They are just used as wires for PCIe lanes

[@also5015]

@@ArtofServer oh wow, so they are interchangeable? when used with PCIe NVMe cards to NVMe backplanes they transmit pcie signals and with SAS backplane/sas controller, it is SAS?

[@Tibbon]

Are there any compatiable backplane options for PCI-e 3.0? For NVMe's I'm thinking of just trying to use individual PCI-E 3.0 cards, although obviously there's a size limit to how many I can use

[@ArtofServer]

The 13th gen Dell servers have a similar kit that is PCIe 3.0. I suspect, though have not tried it, you can fit that kit into a 12th gen server and have it be PCIe 3.0.

[@Tibbon]

@@ArtofServer Thank you! I'm getting my first R820 and still learning about what's compatible, what's undocumented, and what can be made to work! I do DevOps for a living, but in the cloud only and haven't done much hardware in 20 years outside consumer PC stuff.

[@touhedrony6504]

Very well done. is it possible to upgrade in Dell PowerEdge R740?

[@ArtofServer]

Thanks. Sorry, I don't know much about R740. There is a similar U.2 NVMe upgrade kit for the R730 I know of, but not sure it applies to R740. I imagine Dell has similar options for R740, just cannot confirm. Thanks for watching!

[@touhedrony6504]

@@ArtofServerThanks

[@Maxx-qq8lh]

What I do know is that the R740xd supports many more SSDs natively inside its backplane (I think up to 24). The R740 backplane doesn't seem to support NVMe (U.2/U.3) drives. See the Technical guide for the series.

[@Maxx-qq8lh]

OK to set this straight - R740xd (only that model's backplane) appears to support many (up to 24 ?) NVMe drives (U.2/U.3). The plain vanilla R740 doesn't seem to support any of these - according to the "Technical guide".

[@davidfarning8246]

Sorry if I am late for the party... I appreciate your introduction to the world of used Dell servers! Do you happen to have a copy of the spreadsheet you were using to plot out the results of the speed tests? I recently got an R730 and am doing a series of benchmarks as I experiment with ZFS and different cache sizes and cache drives.

[@ArtofServer]

Sorry, I don't have those spreadsheets posted publicly. Glad you find my videos helpful!

[@Jurka666]

Hello art of server! many thanks for this video. i have a question, if i have R820 with 8 sata/sas hdd bays, why there is no such hdd case for the 8x NMVE drives? when i install case for 4x NMVE , i lose 4 another slots? is it right? is there any case for 8 NVME drives? Thank you

[@meateaw]

You'd need a backplane that added an additional 4 NVME connectors to take up another 16x pci-express slots. I suspect they just don't offer it.

[@LuK01974]

wrong pcie card the model you need parts 9460-16i

[@ArtofServer]

Have you confirmed the tri-mode firmware's NVMe mode will work in this case? I've read that the NVMe connection requires special cables. How would you find a special version of those cables to go from SFF8643 to the SFF8087 of the Dell U.2 backplane? And does the U.2 backplane in the Dell work with that?

[@LuK01974]

@@ArtofServer same cable noting fancy the problem is the price of this controller (600/1000) but you able to find already cheaper with adaptec controller. The goal il 12Gb/s not 5 is not "legal" in this build!!! :-D

[@LuK01974]

try with HP H240 SAS-3 I see you have one in your ebay i'm shure working perfect.

[@ArtofServer]

@@LuK01974 that H240 doesn't support NVMe.

[@LuK01974]

@@ArtofServer after a little research and a call to my dell man yeah not good and not good already cable you use. You need a controller and cable. With same controller 9460-16i you able to control all other disk's you have and improve performance. He told me Luca keep attention to the temperature to the controller if is possible add a little fan.

[@stefanossteff4805]

Hello, Have you tried to add the network card "Dell Ultra-Speed Drive Quad NVMe M.2 PCIe x16 Card 4x1tb" to server r720?

[@ArtofServer]

have not. but i will be adding something like that to my Z840 in a video soon.

[@Maxx-qq8lh]

It's very unlikey to work in an R720 as there's no PCIe switch chip (PLX or otherwise) on the Dell Card you mentioned. That DELL card needs a mainboard with PCIE Bifurcation support that the R720 / R820 do not provide...

[@Eslam.Basyouni]

Thank you, i have Dell R720 with SAS HDD , i want to know if the Normal SSD can work on it or No, and from other side how can i check the SAS Health while it work in Server RAID

[@ArtofServer]

what is a "normal SSD" ? SMART data can be gathered off the drive, even from a RAID controller. That should provide drive health data.

[@qnax]

Video beast... What is the name of this program that is monitoring the I/O in the right frame? Of the ones I found here all getting refreshed, the one you're using seems to be runtime.

[@ArtofServer]

iostat

[@Arek_R.]

Last time when I worked on a server was 9 years ago when I was 17yo, yet I can see that you're supposed to remove the backplane to install the cage because I can see solid dents in the blue plastic that you made by trying to smash it in with backplane on it, after you got it installed it also clearly looks like it would never fit with the backplane on the cage. And so you do not need to remove the trays nor the blank. Next time if you see any blue plastic on any HP/Dell machines it probably means that you have to do something with those to install/uninstall stuff. I had two engineers try to take the CD drive from a small PC, they couldn't do it, then I came and saw the front grill piece had two tabs with blue dots, pulled on those and entire front grill came off and it was needed to remove the CD drive, they looked at me like I'm a wizard.

[@ArtofServer]

Well, I lived and learned... :-)

[@xtremeownagedotcom]

I am rather curious to know why the performance seems so low on large sequentials. I would expect.... much higher throughput from U.2 NVMe.

[@Maxx-qq8lh]

Well if you watched the video fully you would know the answer - the "Native" kit for the R730 (Gen12) Servers runs at PCIe 2.0 (!!) speed only. So faster drives (supporting NVMe PCIe 3.x) would be driven with PCIe Gen 2 speed. Not good.

[@Maxx-qq8lh]

Sorry ... I meant the "native" DELL kit for the R720 / R820 of course.

[@FlexiK4LP]

Hi there, i didn't get the point when you get zfs running on a centos7 os? Also the iozone is not installed from official sources or?

[@ArtofServer]

yeah, I usually just use my portable CentOS 7 install on USB. I show how to do this in this video (kzbin.info/www/bejne/opKmaKGpjpaGr68) Nothing special. And yes, i don't think iozone is in the official repos. But the iozone project does provide RPM so it's easy to install.

[@montiedj]

I have proposition, can You test supported SSD for perc controlers with consumer SSD. This is a big problem, enterprise SSD are very expensive and i know they have a high enurance for working. A consumer SSDs are great too and they have a 1/3 price enterprise. I have tested a few ssd and they work in PERC H710,H700.

[@ArtofServer]

I agree with you about cost when buying brand new SSDs. But you can buy enterprise grade SAS-2 and SAS-3 SSDs for pretty affordable prices on 2nd hand market in USA at least.

[@montiedj]

@@ArtofServer Unfortunately i am from Europe Poland. Yes we can buy used ssd but sometimes new ssd from vendor are working great with dell perc raid controller. The are many posts in dell forum with asking people about supported ssd. For example kingston a400 ssd worked fine but in specific firmware(from end 2019year manufactured), dell ssd from pc or laptop(dell branded ssd from samsung). Maybe somebody can create a white list these ssd

[@χρήστοΣταύρου]

Can i use an r820 as a workstation?

[@ArtofServer]

maybe a headless workstation, but I don't think it would be useful as a normal workstation. Dell does sell rackmount workstations, so perhaps one of those would be better suited.

[@irvingaguirre21]

Hi Art of server, im have some doubt about a controller and a server i have a supermicro server with a SAS-846EL1 backplane with a 24 disks support , planning to do a RAID6 with 24 x 8 TB The Supermico Mobo is a X8DTN+-F actually the server has a Dell perc h700 with 8 x 8TB Iron Wolf on a RAID 0, Which card do you recoomento to support the 24x8 with this backplane ?

[@ArtofServer]

That backplane needs 2x SFF8087 connections. You can get by with 1, but for 24x HDDs, I think 2 is better. So, you have a RAID card (H700), are you looking for a different RAID card or a HBA card? What "doubts" do you have?

[@irvingaguirre21]

@@ArtofServer Thankyou for the response. The card actually presents problems, when the server restarts is neccesary to unplug and plugin again, because the bios don´t recognise the card, so i need to change it. The h700 has a 2x SFF8087 connections, if i get a new H700, can support the 24 disks, or which card that you sell do you recommend ?

[@The0Kuki]

I would like to use a U.2 drive in an PCIE adapter in R720. Do you think that this is possible?

[@ArtofServer]

Yes, it is possible as a PCIe storage device, but you will not be able to boot off it directly because these systems did not support NVMe booting. There are workarounds to that problem, but it is not possible by default.

[@The0Kuki]

@@ArtofServer thanks. I will try that soon. Any chance you have link to that workaround?

[@ArtofServer]

@@The0Kuki there are several ways to workaround the problem, but one way with Linux is demonstrated in my video here: kzbin.info/www/bejne/iaakn2l_fJmefJY the other ways involve firmware modifications to the motherboard.

[@newptone]

R820 supports PCIE Gen 3, while the expansion card only supports Gen 2. I just hit the same issue and find Dell's reply: www.dell.com/community/PowerEdge-HDD-SCSI-RAID/R820-Gen2-PCIe-NVMe-Card/td-p/7796487

[@morosis82]

I know this is a tad older now, but looking at the 'unexpected discovery' part, the card itself is only x8, so max of x8 PCIe3 capable. Given that it supplies 16 lanes to 4 devices, they're only going to be able to do a max of PCIe2 speeds anyway. Probably why they use that chip, and why the cage only supports 4 devices.

[@ArtofServer]

The PCIe switch card is x16 on the host side, and 4x4 on the U.2 side. The system is PCIe 3.0 capable and has a x16 electrical slot. So, there's enough bandwidth between the host and the NVMe drives, but that PCIe switch is not fast enough. This was first gen NVMe, so maybe the drives from that time period weren't that fast - idk... but the setup could benefit from a PCIe 3.0 capable PCIe switch (and Broadcom does make them).

[@alleyezondisromo]

@@ArtofServer So I am a bit confused with this card, because i am trying to set a setup like this on an r720. I actually bought the GY1TD (which is the pcie 3.0 card for the r630), and im still waiting for it. I guess im most worried now that I wont be able to get the 4x4 bifurcation that i am looking for. Are you getting 4x4 bifurcation using this HBA?

[@ArtofServer]

@@alleyezondisromo I assume you didn't mean HBA but the PCIe expansion card I used in this video? If so, that card has a PLX PCIe switch chip on it, so it doesn't require bifurcation.

[@johnscully1924]

I am running R730s (which will do bifurcation if you upgrade the firmware), with Dell quadspeed Nvme cards in a PCI3 X16 slot. 4 Samsung EVO970 drives. Those drives do seq read of 3,500 MB/s and seq write 2,500 MB/s They perform at full speed. I think you are being more limited than you think by the fact that it is a PCI switch without bifurcation. That is only a guess...but it would be very interesting to see you run exactly the same tests with faster drives on an R730 quadspeed card

[@ArtofServer]

I think the DC P3600's seq read is being limited by the PCIe 2.0 x4, but not the writes. I don't think the PCIe switch (other than it being 2.0) is the limit vs a x4x4x4x4 bifurcation, as the single drive vs 4xRAID-0 show basically a linear scale up. Of course, 4x 970EVO+, at PCIe 3.0 x4 will be awesome! But keep in mind, I don't think that has any kind of PLP, not in the M.2 SSDs, nor the PCIe x16 adapter card. The only one I have found that has PLP is the HP Z TurboDrive Quad Pro. So, as fast as it is, have to be careful about using it for something like ZFS. The performance of the 970EVO/+ is due to it's large SLC cache and slows down to about 1700MB/s when you exceed that cache.

[@johnscully1924]

@@ArtofServer a bit more info: I run 6 servers running proxmox virtualization with CEPH distributed file system. Hyperconverged high availability cluster. Dual power feeds with independent power feeds and generators so I am not as concerned about things like PLP. I have redundancy in the file system as well. I think samsung is about to ship an m.2 with PLP - it is longer than the evo 970, so may not fit all cards, but it is a step in the right direction Just jumping from 10G to 100G networking. I wish there was a third PCI3 x16 slot in the 730. One for the quadspeed card, one for the 100G card takes the two there are.

[@johnscully1924]

@@ArtofServer I should check...if the nvme cage for the 730 gives the same performance as the quadspeed card, I could use that...use the new 983 DCT's NVMe which does have PLP I think sorry - the 883 is the U.2 The 983 is a m.2 with PLP www.samsung.com/us/business/products/computing/ssd/enterprise/983-dct-1-9tb-mz-1lb1t9ne/b2bapp/

[@AchwaqKhalid]

1st 💪

[@shephusted2714]

pci gen 3 is not present until next model r830 so you upgraded for nothing essentially - you can't get the best speed - refurbs

[@ArtofServer]

R820 is PCIe 3 system, but the PCIe switch on that card is PCIe 2.

[@kristopherleslie8343]

Is this possible with R710 also?

[@ArtofServer]

No, I would not recommend it because R710 is a PCIe 2.0 system and that will bottleneck NVMe drives. Better to upgrade the server.

[@kristopherleslie8343]

@@ArtofServer have been playing with 3 R210 iis and 2 r710s. Any life left in there? I have an extra r540 but it’s a temp server

[@ArtofServer]

@@kristopherleslie8343 sure, I think they are great machines for certain purposes.

[@kristopherleslie8343]

@@ArtofServer i was thinking of just using thr R540 for the storage and then pass it out to the other host. I’m using XCP-NG and even Proxmox but mostly local storage. I’ve seen the lack of iops :(