SI is not so god in sys admin stuff. It can do basic stuff. But it fails in more advanced stuff not written on forums.

Yet another deepseek video on the platform, but it helped me get my voice2text transcribe app going, which is nice ~ Oh, and I forgor to ask, like & subscribe!

Hey I don't suppose this could be to do with LLDP advertising PAUSE flow control capability?

Came here from the X710 video. Now I'm remembering substation alpha and aegisub

I expect "no one will have to scratch their head this hard" will be repeated in every video. Very subtle.

Arghhh LLDP, good on You sir, I´m happy you got the root cause.

I have that exact same switch

BSD people can set the tunable as: dev.ixl.0.fw_lldp -> set to "0" Subsequent cards / ports would be dev.ixl.1.fw_lldp, dev.ixl.2.fw_lldp,...

These issues would be a lot more valuable use of AI vs. asking it how to write my term paper. Self configuring printers and presentation/display/VDI tech would also be arguably as value, if not more so.

I’m following this like the latest episode of Dexter.

You earn my highest respect. I wish I had a person like you to work with. Kudos from Victoria, BC, Canada.

The firmware lldp agent caused a lot of issues at my previous job, not just with LACP or MLAG. The issue we were seeing is that the firmware agent only supports MAC addresses from EEPROM, not from UEFI. So we could easily see if a box had booted to userspaxe, it’s lldp ID changed from 00:00:10:00 to a proper ID. And if you’re upgrading to E8xx, enable DCB in the linux kernel for lldp to work…

I'd love to learn more about the sas/FC ethernet bridging devices. I actually was just tinkering with an old Dell server to hook up to a big dumb SAS hdd enclosure to do iscsi.

It is possible to turn off the "unsupported" SFP check by use of the firmware tool -- and also enable use of other optics such as LR, which is also disabled from the factory on the x710.

WTF intel

Hi. On our Cisco Switches LLDP Packages can also be turned off on the switch side. So maybe your port configurations on your switches is just not the same hence the different behaviour?

Glad I only use switch independent teaming. My X710s still work great, but my HP labeled Broadcom 530T immediately crash with the same configuration as soon as a second link is added. Kind of weird…

"Dell storage only works with Dell branded drives" My experience is that other drives work fine, BUT it will show a warning for the drive that you can't dismiss. Somehow the service contractor sent us a replacement disk once that wasn't listed as a tested/compatible drive. This was for an EqualLogic 6500. We left it in, and never had any issues with it. Of course YMMV.

ehm did you try to update the firmware?

great video but it seems to me the whole point was to pat yourself on the back for a great job just my 2 cents thank you great job heheheheheheh

I just upgraded to dual 25GbE networking from an Intel dual 10GbE card. I did a RAM drive to RAM drive performance test on my old 10GbE's with bonded interfaces and was only ever able to achieve ~16gbps no matter what settings I screwed with. I was using 3rd gen AMD CPUs, so it shouldn't be PCI-e bandwidth limitations. Turns out the cards are PCI-e Gen 2, but that still should have been more than enough bandwidth at 8x links. Maybe it was the backwards compatibility overhead or something at that speed with the newer CPU/mobos? The world may never know. I still need to do a real bandwidth test on these new Gen 3 25GbE cards I have bonded now.

I had an identical issue with a 4x10gbit LACP bond between a Proxmox server with a XL710 and a Mikrotik CRS328-24S+2Q+. Upload stuck on 6mbit. Solution was not to disable LLDP entirely but only turn off LLDP DCBX on the switch side.

I was going to but now that you told me to, I'm not gonna

lol is that linus clip a deep fake or was he actually on tv? xD

I work for a vendor that previously used the X710 in their appliances. Years ago our appliances were experiencing kernel panics and we found the X710's i40E driver's handling of LLDP DCBX frames to be the culprit. Unfortunately the guidance at the time was to disable DCBX or connect the ports to a switch that doesn't utilize DCBX.

With the XXV710 or whatever i debugged this. There Its a dcbx sub tlv for virtualization for EVB or VEB where the upstream switch directs the card to do things. Sending that tlv but no instructions fucks with the cards offload brain. Yours is different but will be along the same lines. Get me a 10g ethernet tap and a bunch of nexus or your acx's and those cards and i can likely prove it 😂 But i know its true anyway. I can make you a video with the more specific explanations. You can probably also prove it yourself if you manually set the tlv's the switch sends out to a most minimalistic set, i.e Just the port desc

The x710 is the worst NIC I have ever personally used. It’s soured me on Intel completely. It’s a shame because the x510 was a proper workhorse NIC. It’s got an astonishing number of firmware and driver bugs. You’ll almost certainly hit one or three deploying them. These days I stick to Broadcom generally. I’m not sure how Mellanox is beyond Nvidia doesn’t need any more money. 😅

Hello fellow academic HPC sysadmin. Looking forward to your SC24 video, I am curious what you thought of this past conference compared to the past few years. Did you see a company is offering proxmox support and promoting it on the show floor? At 2023 there was a company showing off shared ram systems that I was excited about but I did not see them this year. Leaving this note so the algorithm might promote you a little more. We need more research academia sysadmin content.

So TLDR, Intel X710 + LACP w/ LLDP is bugged

Yeah it's a popular NIC. They are popping up on eBay now. I've also seen it integrated on motherboards. You'll end-up helping a lot of people.

no such option in Windows 2019 Server under advanced settings for the Intel 710 fiber NIC.

i have a x710 on my proxmox system aswell and for me i have this weird issue where i can recieve with 10gig but sending is limited to around 6gbit with iperf3 or also inside a vm copying big files over the network. anyone have any idea what that could be?

There are other use cases outside of LACP where you need LLDP in the kernel and user space, so many thanks to hunting this down to its real root cause!

LLDP causes your problem because it changes your (default) source MAC address on both ports of the dual port 10Gb/s NIC to the same MAC address (that of the bonded interface). Effectively the MAC address will move with each LLDP announcement. That's a problem with active-passive failover bonded pseudo-interfaces.

Worked in a DC for a global cloud provider for 3 years, and our primary cards were Intel x710s (up until Intel released SapRap CPUs), or where those weren't available, their x540s. Never encountered this issue, but we were flashing BIOS as soon as the servers were racked, and the DC these issues would be tested and found in was in another country. Interesting as hell to see that this was an issue on such a widely deployed chipset (and I can't help but wonder if it's also dependent on the card/device it's used in as well)

Sharing is caring! 🦾

So I have an "Intel(R) 82599 10 Gigabit Dual Port Network Connection" NIC in my Windows 11 PC. How do I disable LLDP? I looked in the device manager but can only find settings like Jumbopackages, large-send-offload v2, UDP checksums etc. Where do I find LLDP?

Like for UNATCO theme.

This dude never blinks. 😯

"In the enterprise world, it's actually quite common for vendors to lock hardware" - yeah, and that really needs to change. Not sure if we need legislation or more Yoshis, but the extreme corporate greed has to end.

LACP is quite neat but implementations differ. LLDP and its extensions are very vendor specific. It may be that you have played around dynamic LACP - static one doesn't need supplimental information provided by LLDP

Did you also check for latest firmware?

I also had issues with a broadcom 10Gbit NIC.. So you're not alone ;P

I think its a 30cm problem

Why would Linux insist on using LDP in order to enable LACP? Can't you force it? Sadly it's not the first time I encounter networking weirdness within Linux.

People spending a week of their time to troubleshoot a problem and then go back and post the technical details everywhere they can is why the tech world continues to function. Thank you!

Yeah I think that's what brought me here, my Minisforum MS-01 has 2 X710 interfaces. They seem extremely common.

the intel 82599es are better and less troublesome.

Ha, x700 family well know as the crappy one. In my practice, the support preferred to just replace to the different brand rather than deal with it to same cycles and the cost. Despite all features advertised by Intel, it worked well only when they were mostly disabled.