Turn off this option if you have 10Gbit Intel NIC

[@DCTekkie]

Remember you can suggest me more things to do a vid on on my discord! discord.gg/W385Wu2Z

[@nou712]

Why are you locked in the bathroom? (Why are you using Deus Ex UNATCO ambient soundtrack in your video?)

[@DCTekkie]

That's because I'm a gigantic manchild weeb. I also use Evangelion soudtrack and Wild Arms 5 in this vid :D

[@bmenrigh]

People spending a week of their time to troubleshoot a problem and then go back and post the technical details everywhere they can is why the tech world continues to function. Thank you!

[@snemarch]

"In the enterprise world, it's actually quite common for vendors to lock hardware" - yeah, and that really needs to change. Not sure if we need legislation or more Yoshis, but the extreme corporate greed has to end.

[@Javanoiac]

Thank you so much. Unrelated to LACP but I was having serious issues with my single onboard 5GbE Realtek NIC on my desktop PC (Win10 OS) and I tried every possible thing that I could think of except disabling LLDP for the interface (never even crossed my mind). Watching your video gave me the idea to try and it worked like a charm. I'm using a 1GbE home switch and my onboard NIC would consistently cap the link speed down to 100Mbits or occasionally shut down the interface itself completely. It was practically unusable for me so I used an external ethernet adapter as a temp solution for some time. I had never encountered this particular issue or any LACP related issues with servers or enterprise network hardware before. This has been an experience. Thanks again.

[@magran17]

You earn my highest respect. I wish I had a person like you to work with. Kudos from Victoria, BC, Canada.

[@Haplo164]

These probably go more views because a lot of home labs have Intel 10 gig NICs and they all look the same. I have a few x520s and I thought I might pick up something that would apply to those and/or hear about a sweet piece of cheap hardware. Either way it was worth the watch.

[@DCTekkie]

I'm glad you found something interesting!

[@thomaslindell5448]

Exactly applies to me I run a vrtx chassis with 4 m640 blades. And use x510 dual port nics

[@banggugyangu]

I have a couple Intel x520s in my home lab, myself. Don't currently have them bonded, but I'm intending to soon.

[@stringsofnihility]

I work for a vendor that previously used the X710 in their appliances. Years ago our appliances were experiencing kernel panics and we found the X710's i40E driver's handling of LLDP DCBX frames to be the culprit. Unfortunately the guidance at the time was to disable DCBX or connect the ports to a switch that doesn't utilize DCBX.

[@BestSpatula]

But why does LLDP cause this problem?

[@DCTekkie]

Supposedly Linux and VMware networking stack depends on lldp packets for lacp to work. With lldp agent enabled in the firmware, lldp packets never reach Operating system. This is the explanation I found but couldn’t confirm 100%

[@zviratko]

@@DCTekkieI don’t think LLDP is the cause, it likely has some side effect that resolves the real cause. What LACP policy did you configure on the host and the switches? What information does your switch’s LLDP contain? It could be something as simple as LLDP originating from the wrong MAC after LACP is configured.

[@KoenKooi]

The firmware lldp agent caused a lot of issues at my previous job, not just with LACP or MLAG. The issue we were seeing is that the firmware agent only supports MAC addresses from EEPROM, not from UEFI. So we could easily see if a box had booted to userspaxe, it’s lldp ID changed from 00:00:10:00 to a proper ID. And if you’re upgrading to E8xx, enable DCB in the linux kernel for lldp to work…

[@autohmae]

4:42 THAT is something I did not know at all, well both things in-build LLDP engine and the LACP relation.

[@crestdazoltral7705]

LLDP causes your problem because it changes your (default) source MAC address on both ports of the dual port 10Gb/s NIC to the same MAC address (that of the bonded interface). Effectively the MAC address will move with each LLDP announcement. That's a problem with active-passive failover bonded pseudo-interfaces.

[@adrianbool4568]

I think you're mixing up LLDP with LACP....

[@JamesTK]

@@adrianbool4568they're not, the issue was with the firmware handling lldp when lacp was enabled

[@liewchengyeh]

Intel can't seem to fix all their Base-T Ethernet Based NIC (ever since their 1GBe, 2.5GBe or 5GBe.....) Every chip must have a least one or some minor driver/hardware issue....

[@Ninos.D]

I225/i226... I have the i225 v3 and it's a piece of shit. Will never buy a motherboard with integreted Intel NIC. I use a USB adapter with Realtek and works like a charm.

[@ws_stelzi79]

Who wouldn't like to have a few datacenters in their homelab? 🤪🤔

[@guytech7310]

Nope. With a home DC comes a monster power bill! With electricity prices on the rise, I spec new servers based upon power usage, to avoid those high power bills.

[@stevesteve8098]

@@guytech7310 Yep .. I used to run a few gaming servers for the community... but now it is impossible due to power costs.

[@zarkeh3013]

those who have to pay the power bill might

[@vabello]

@@guytech7310 I used to work at a small DC, and I think the power bill was around $30,000 USD a month.

[@autohmae]

Tip: you really don't need to keep it at 18 C degrees, you can easily choose 4 degrees higher and still be fine (maybe turn off the AC to see how long it takes to go from 18 to 22 to confirm it doesn't add much time in case of a AC outage). You should safe a bunch of energy that way.

[@TheDiveO]

There are other use cases outside of LACP where you need LLDP in the kernel and user space, so many thanks to hunting this down to its real root cause!

[@ChristianHitchcock-s7p]

I was going to but now that you told me to, I'm not gonna

[@vadimesharak726]

Ha, x700 family well know as the crappy one. In my practice, the support preferred to just replace to the different brand rather than deal with it to same cycles and the cost. Despite all features advertised by Intel, it worked well only when they were mostly disabled.

[@joshuawaterhousify]

Worked in a DC for a global cloud provider for 3 years, and our primary cards were Intel x710s (up until Intel released SapRap CPUs), or where those weren't available, their x540s. Never encountered this issue, but we were flashing BIOS as soon as the servers were racked, and the DC these issues would be tested and found in was in another country. Interesting as hell to see that this was an issue on such a widely deployed chipset (and I can't help but wonder if it's also dependent on the card/device it's used in as well)

[@barmazu]

Yup, as always the smallest switch on the panel 🤡 Troubleshooting well done, sir.

[@DCTekkie]

Thanks! I have to say, google being better at googling (or me doing more thorough search from the get go) could potentially save me from making two videos on this

[@autohmae]

You might want to read the article: the men who killed google

[@mstandish]

Hello fellow academic HPC sysadmin. Looking forward to your SC24 video, I am curious what you thought of this past conference compared to the past few years. Did you see a company is offering proxmox support and promoting it on the show floor? At 2023 there was a company showing off shared ram systems that I was excited about but I did not see them this year. Leaving this note so the algorithm might promote you a little more. We need more research academia sysadmin content.

[@DCTekkie]

Totally agree we need more proper sysadmin content! The SC24 video is out, i've started attending SC since 2021 (first one after covid) and every year it just keeps getting bigger. I think I know which company you're talking about, I think they also promoted ceph support. I also think you need to attend at lest 2 or 3 to get an idea how do trends work in HPC enviroment. For me, this year it was interesting to see how quantum took a step back and cooling solutions grew in its place. Also there was a booth where they started doing computations with laser beams, possibly shifting to a new paradigm of computing

[@rabidpb]

It is possible to turn off the "unsupported" SFP check by use of the firmware tool -- and also enable use of other optics such as LR, which is also disabled from the factory on the x710.

[@chbrules]

I just upgraded to dual 25GbE networking from an Intel dual 10GbE card. I did a RAM drive to RAM drive performance test on my old 10GbE's with bonded interfaces and was only ever able to achieve ~16gbps no matter what settings I screwed with. I was using 3rd gen AMD CPUs, so it shouldn't be PCI-e bandwidth limitations. Turns out the cards are PCI-e Gen 2, but that still should have been more than enough bandwidth at 8x links. Maybe it was the backwards compatibility overhead or something at that speed with the newer CPU/mobos? The world may never know. I still need to do a real bandwidth test on these new Gen 3 25GbE cards I have bonded now.

[@DCTekkie]

@chbrules maybe you used wrong slot on mobo?

[@Joachim1292o032043op]

2:29 The whole country is running on gbc photonics SFPs XD

[@DCTekkie]

Shhh don’t let them know the gbc also sells optics reprogrammer ;)

[@ErikBussink]

Good video and good troubleshooting. I would not have found in my own HomeDC because, I'm not using LACP with my ESXi hosts for 15+ years thanks to the networking stack in the ESXi hypervisor and how it can use multiple NICs. (I'm not trying to start a Proxmox vs ESXi thread).

[@GerbenWijnja]

"Dell storage only works with Dell branded drives" My experience is that other drives work fine, BUT it will show a warning for the drive that you can't dismiss. Somehow the service contractor sent us a replacement disk once that wasn't listed as a tested/compatible drive. This was for an EqualLogic 6500. We left it in, and never had any issues with it. Of course YMMV.

[@VTOLfreak]

I had an identical issue with a 4x10gbit LACP bond between a Proxmox server with a XL710 and a Mikrotik CRS328-24S+2Q+. Upload stuck on 6mbit. Solution was not to disable LLDP entirely but only turn off LLDP DCBX on the switch side.

[@TheNefastor]

Yeah it's a popular NIC. They are popping up on eBay now. I've also seen it integrated on motherboards. You'll end-up helping a lot of people.

[@barrymerritt3727]

I always considered LLDP and CDP to be a security vulnerability that is problematic. LACP should be limited to the layer 2 switch interfaces and negotiate directly with the NICs without any broadcast across the network. Intel's reliance on LLDP for LACP is a bug, not a feature.

[@autohmae]

You did not hear it correctly, he said Linux and VMware depend on LLDP for LACP. Also: LLDP is NOT forwarded by any LLDP capable switch, only stupid switches like undamaged switches forward LLDP to other ports.

[@udirt]

There have been sec issues here and there, mostly it's information disclosure and for that you can just use it right. I.e. on a dmz switch don't receive LLDP from the hosts and don't send the management address. The hostname is info too, yes, but the value to ops exceeds the value to attackers. And noone stops us from having a different scheme for DMZ except lazy coworkers. And those are the bigger concern 😂

[@OsX86H3AvY]

ive got an x710-da4 in my pfsense router/firewall going to an x520-da2 in y proxmox server and it runs great. I also have an x710-da4 in that server (so, 6 10G ports in total) but I havent tried lagg'ing from the x710 in the proxmox box to the one in the pfsense though...and I'm switching out to 25G in this next weel anyways so i wont really care but its good to know....ive heard these X710's have given folks troubles but I've never had probs with them....nice vid keep it up pls

[@DCTekkie]

You could try going for 100G NIC that can break out into 4x25 ;d

[@peterpain6625]

@@DCTekkie Try to put the Cisco branded ones on a PCI-E rev3 x8 for "funny stuff to happen" ;)

[@michaelkreitzer1369]

The x710 is the worst NIC I have ever personally used. It’s soured me on Intel completely. It’s a shame because the x510 was a proper workhorse NIC. It’s got an astonishing number of firmware and driver bugs. You’ll almost certainly hit one or three deploying them. These days I stick to Broadcom generally. I’m not sure how Mellanox is beyond Nvidia doesn’t need any more money. 😅

[@brylozketrzyn]

LACP is quite neat but implementations differ. LLDP and its extensions are very vendor specific. It may be that you have played around dynamic LACP - static one doesn't need supplimental information provided by LLDP

[@Taverius]

Yeah I think that's what brought me here, my Minisforum MS-01 has 2 X710 interfaces. They seem extremely common.

[@MelroyvandenBerg]

I also had issues with a broadcom 10Gbit NIC.. So you're not alone ;P

[@MelroyvandenBerg]

Turned out the card was not getting enough air cooling, causing the card to become too warm (over 95 degrees Celsius) hehe

[@klauserwin9860]

This dude never blinks. 😯

[@comesignotus9888]

Interesting, good to know about this peculiarity of Intel cards. Thank you!

[@MelroyvandenBerg]

Did you also check for latest firmware?

[@antonlonnerbro1842]

Love the hoodie, did you make the design yourself?

[@lertsek101]

Hi. On our Cisco Switches LLDP Packages can also be turned off on the switch side. So maybe your port configurations on your switches is just not the same hence the different behaviour?

[@jensdroessler3575]

If this is a known problem, why doesn‘t Intel fix it in firmware or why isn‘t it deactivated by the driver?

[@DCTekkie]

@@jensdroessler3575 from what I found they decided to keep lldp on by default and added the flag support to i40e module so you could in theory manipulate the setting with ethtool

[@jensdroessler3575]

@ Yes, but other NICs surely that way, too, aren‘t they? And it seems to work there…

[@andrewenglish3810]

no such option in Windows 2019 Server under advanced settings for the Intel 710 fiber NIC.

[@DCTekkie]

I've set it in card's firmware, depending on your platform you may have this setting in your BIOS

[@cyklondx]

so, i have 10gig fs sfp's for dell branded intel side, and cisco switch side - i've fixed exact same issue by upgrading firmware on sfp modules. Once that was done it would show all details on idrac for the sfp modules - what they are connected etc, it would then pass it by to os and everything would work...

[@studioxxswe]

ACX7024 in a homelab? :)

[@DCTekkie]

I might just roll with this and pretend this is a homelab ;)

[@ErikBussink]

@@DCTekkie That is also why I'm calling mine a HomeDC and not a Homelab ;-)

[@ThePlumbeus]

Glad I only use switch independent teaming. My X710s still work great, but my HP labeled Broadcom 530T immediately crash with the same configuration as soon as a second link is added. Kind of weird…

[@berndeckenfels]

Whats the 6GBiTS cap it „activates“?

[@plebbyplebster9595]

Thumbs up just for the deus ex theme alone

[@BestSpatula]

I love this game.

[@mrlithium69]

So TLDR, Intel X710 + LACP w/ LLDP is bugged

[@__Mr.White__]

So I have an "Intel(R) 82599 10 Gigabit Dual Port Network Connection" NIC in my Windows 11 PC. How do I disable LLDP? I looked in the device manager but can only find settings like Jumbopackages, large-send-offload v2, UDP checksums etc. Where do I find LLDP?

[@maxiosu]

It's part of the settings of the network connection like ipv4/ipv6 address config. "Microsoft LLDP Protocol Driver". If you have played with Hyper-V/WSL that may have activated it similarly to VMware in the video.

[@__Mr.White__]

@@maxiosu Thanks Mister Maxi 👍

[@jorisdevaan6845]

ehm did you try to update the firmware?

[@BadMax02_VR]

i have a x710 on my proxmox system aswell and for me i have this weird issue where i can recieve with 10gig but sending is limited to around 6gbit with iperf3 or also inside a vm copying big files over the network. anyone have any idea what that could be?

[@Olivyay]

Why would Linux insist on using LDP in order to enable LACP? Can't you force it? Sadly it's not the first time I encounter networking weirdness within Linux.

[@BersekViking]

Thank you!

[@raulmiguelramos]

Thanks, for sharing

[@DCTekkie]

Thanks for watching!

[@peterpain6625]

Interesting. Thanks for sharing

[@wildorb1209]

There's obviously no cable management issue at work for you... If my people worked like that, they wouldn't be with the company for long.

[@DCTekkie]

When a datacenter is almost 30years old some mess tends to form and stay ;)

[@DavidStephens18]

Get education to pay for neat and tidy and sure it will be done but they dont care if it works

[@ildarakhmetgaleev]

Like for UNATCO theme.

[@stevesteve8098]

the intel 82599es are better and less troublesome.

[@billkillernic]

lol is that linus clip a deep fake or was he actually on tv? xD

[@DCTekkie]

He was, check out "Linus tonight show" ;d

[@billkillernic]

@@DCTekkie Nah this probably would worsen the situation that noob needs to be defamed not boosted up xD

[@NetrunnerAT]

I think its a 30cm problem

[@udirt]

With the XXV710 or whatever i debugged this. There Its a dcbx sub tlv for virtualization for EVB or VEB where the upstream switch directs the card to do things. Sending that tlv but no instructions fucks with the cards offload brain. Yours is different but will be along the same lines. Get me a 10g ethernet tap and a bunch of nexus or your acx's and those cards and i can likely prove it 😂 But i know its true anyway. I can make you a video with the more specific explanations. You can probably also prove it yourself if you manually set the tlv's the switch sends out to a most minimalistic set, i.e Just the port desc

[@ankantan]

🎉👏

[@rolandcollins1427]

great video but it seems to me the whole point was to pat yourself on the back for a great job just my 2 cents thank you great job heheheheheheh

[@adrianbool4568]

I think is point was to assist others should they find themselves with the same problem → they can get a headstart by looking at their LLDP config...

[@usermanico]

WTF intel

[@spx2327]

Sharing is caring! 🦾