AI Networking is CRAZY!! (but is it fast enough?)

[@NetworkChuck]

Learn more from Juniper Networks on how the network will save the AI data center: juniper.net/AIwars Dive into the heart of the AI networking battleground where the demand for speed and efficiency tests the limits of technology. This video unravels the complexities of training AI models, highlighting the network bottlenecks that slow down progress and how innovations like RDMA, RoCE, ECN, and PFC are pivotal in the quest for seamless AI development. We explore the clash between Ethernet and Infiniband, dissecting their strengths and weaknesses in handling the massive data and low-latency requirements of AI training. 🔥🔥Join the NetworkChuck Academy!: ntck.co/NCAcademy **Sponsored by Juniper

[@perq_hvh]

i will be the networkjustin of albania

[@r03irpokwefpojewk]

/

[@andrewvickers6847]

Juniper is being acquired my HPE

[@TheSensationalMr.Science]

I would be curious if they have *load-balancing* also available or if that is considered 'not available' due to the fact of so many GPU's running in parallel and every line is already hyper-saturated. though *ROCERDMA* [sounds like another DMA for net], *Explicit Congestion Notification* and *Priority Flow Control* are interesting. Hope you have a great day & Safe Travels!

[@Litrary-cheese]

6:40 Roblox💥💯💯

[@RylanTech]

I wonder what the security implications of RDMA is since it is bypassing the kernal

[@jonathanrhodes6180]

That was my question to. How can I ensure security to earn money when today's skills are obsolete?

[@steveo6023]

Usually RDMA is used in an isolated environment

[@lubu42]

RDMA is secure in the sense that it's typically implemented on a closed off network that is completely separate from other portions of the network that a supercomputer might be using (I work with supercomputers so that is what I think of when it comes to RDMA). Even with no encryption across the wire, there are keys in place which allow certain NICs and devices to talk to each other so it's a zero-trust configuration. You almost have to treat the high-speed network fabric like it's the bus moving data from CPU0 to CPU1. We have implemented RDMA across two of the supercomputers I work on and the performance is insane, but there still is a potential security risk that needs to be accepted even though the RDMA network is very closed off.

[@hamzagorcevic8443]

@@lubu42im studying computer science , i wish to work with supercomputers or atleast to work as internet eng. but im not sure that my university is teaching me enough about it , can i get some advice how to get job as network eng or to work with supercomputer , anything wich isnt web developing ...

[@MelroyvandenBerg]

kernel*

[@TheNumbersPerson]

Having worked with some of the networking and hardware setups for AI servers... The networking and liquid cooling I got to see there were awesome. Just about anything you can get your hands on is outpaced 10 fold or more in an AI datacenter.

[@silenthill4]

which site code?

[@NoelWillis]

Yep until the tree hugging greenies go on about power usage for cards and data centre cooling

[@AI-PhotographyGeek]

7:39 Someone is behind you man! Just Run 😀

[@jodybrookes]

I saw that too!

[@djangelinfinity]

I got scared lmfao

[@baudosbloomys1590]

What or who was that? Pause at your own risk. That will give you nightmares. I hope you’re still alive!

[@StannyMH12]

Who is that behind you

[@expansivenature]

we need an answer @NetworkChuck who was that weird looking person/creature behind you?

[@davestorm6718]

Anyone old enough to remember running Doom via IPX/SPX (Novell Netware) on a LAN? Remember when TCP/IP was introduced into the stack and slowing the post-Doom games down? Pepperidge Farm remembers.

[@_sneer_]

This whole video must be the most sneaky advertisement I have ever seen in my life.

[@Kazekoge101]

the jump cuts on the other hand...

[@SovietCustard]

It’s not

[@w花b]

Not an advert or not sneaky?​@@SovietCustard

[@TomBuijs]

Hi! Long time watcher of your channel! Ive been a sports instructor for 15 years and ive been offered a job as Junior Network Engineer at a BIG company. I'd like to think everything I learned here helped :)

[@siddharthtadury5612]

Juniper is making more progress than anyone else in the market with respect to AI in the Datacenter environment.

[@bwmacallister]

That thumbnail gets the attention!!

[@adventuretai]

sounds like this is going to introduce a severe set of attack vectors, but none the less its pretty awesome

[@cheema33]

You run the AI training on a dedicated private network. If people are breaking into your private, closed off network, then you have a bigger set of problems to worry about.

[@adventuretai]

@@cheema33 its a new network stack using rdma, it probably won't even be able to communicate with ipv4/6. its going to have a bunch of vulnerabilities and be underdeveloped. Will it get better yes but the main security vector is that rdma. With that said i still think its freaking cool.

[@juz882010]

@@cheema33 Currently they do, but more orgs will want a small setup posible exposed to the internet.

[@Trig0r]

We've just done some testing at work with the guys from Juniper, those switches are beasts

[@siddharthtadury5612]

We are 🙂 Thanks, man.

[@Afran146]

Chuck, your tech videos are consistently awesome and helpful. Keep up the great work! Thanks for sharing your knowledge!

[@ZeroDay204]

1:23 Never thought I would hear Chuck say he wants to become a battery😂

[@thecallofosiris]

What's important is working with what is adopted. As you mentioned ethernet is worldwide and the cheaper option to adopt which means that is what tech's are going to be hired to handle as that is what companies will prefer. This is where tech's can be paid a lot more to apply for positions that will soon open up in connection to supporting infini-band. As IT specialists, we work with companies and their current infrastructure, if the advancement of networking technology is inevitable, we'll all be forced to move along with it!

[@ChrisHorneVL]

As a lifelong Cisco/Arista engineer, I'm so happy that RoCE became a thing. The thought of having to manage a standard ethernet setup and some weird Infiniband tech was soo not what I was wanting to do. Those 800G optics though... not cheap. Dual MPO connections on a single transceiver is pretty cool for breaking out to two 400G ends. I'm assuming we'll have to get some new style of cable/connector soon.. Those OSFPs are monsters.

[@RandomKSandom]

I remember working with Infiniband in 2006. Its cool to see that they're still going.

[@dadamnmayne]

I like that you take your time to make absolutely sure the video is on point before you publish.

[@NoelWillis]

awesome video as always - I am team ethernet as infiband is for a small niche

[@peterwestin1874]

Wow this was cool 😀 I like both of them. A new network system will be needed for Quantum Computers soon as well. Quantum teleportation where the state can not change. Interesting time to be alive . Thank you for this video😊

[@omega.developer]

This is quite an interesting comment. I was exaggerated

[@hawraawakil6027]

Love your channel man

[@Fade2blk9]

Hey Chuck, fellow network engineer here (mostly cloud infrastructure though). First, I loved the video. This helped a ton to clear up my understanding at a high level of InfiniBand. Just some creative criticism, this definitely went much deeper technically than your typical content, so maybe it would be a good idea to state that up front or maybe even break content like this into a separate channel. I would love it if you made more deeply technical content, even beyond this level. You have a great gift for explaining these sorts of things in terms that people can relate to while still keeping it entertaining! Keep up the GREAT work!

[@methuseus]

I didn't even realize it was that deep technically. I guess it just makes sense.

[@liberentis8871]

separate channel, yes more advance would be the best solution

[@BlindPigBluesBand]

Excellent content. What a great introduction to AI Networking. Great work! Thank you

[@最も愚かな]

Awesome video Chuck, you always do an amazing job breaking down explanations. Been subscribed for a while, and I have always noticed very good quality videos. Thank you!

[@Cdaprod]

I work for MinIO and I want to refactor how the world sees and uses its data 🤩 🎉 Thanks for the inspiration :3

[@ecruz69]

Keep Going Chuck !

[@savagepro9060]

Where?

[@c.e.m7535]

Awesome video Bro! Highly educational. Thank you for sharing!🎉

[@legendaryz_ch]

I used to work for a company which built a 76TBit/s ethernet network

[@CODwontons]

Hey great video man - I'm into AI and also out of personal interest I've been learning about networking so this video was awesome and well done

[@GameDr04]

I'm with Ethernet. Sticking with the traffic analogy: we're investing billions in trying to make self-driving cars that function within our existing infrastructure. It's taking a lot of effort, but it will ultimately require less effort, time, and money than to rip out all our roads and streets in favor of a self-driving-specific infrastructure. Using what we have and building within those constraints will require less (if any) overhaul AND the improvements to the stacks will only make things better for everyone, since everyone is already using the base tech.

[@DrewRodgers-wn5ry]

Good stuff ❤

[@jordashthinkosaur9053]

This is the best video I've seen on your channel and I've been watching for years. Your knowledge shone through at an atmospheric level. Please do more like this every so often as the AI landscape broadens!!

[@jannikmeissner]

Fun to see this video right now, as I am planning a big cluster at work. The main argument for me for Ethernet is cost.

[@djdawso]

I'm a long time (now retired CCIE #1937, R/S Emeritus) network engineer and I'm definitely on Team Ethernet.

[@eduardostapenko6808]

1:53-1:54 there's a cool moment... you know what am i talking about :0

[@uskoolkid2007]

Aye Chuck team infi but without the @ and we saw the data center vid! 🎉🎉🎉 thanks for the support (open sourced) infiniband, just like containers/‘Docker; appreciate IT Chuck!

[@nunoalexandre6408]

Love it!!!!!!! Rdma SInce 2019!!!!!!!!!!!!!

[@thebeardofknowledge]

omg, why am I barely hearing about this?! OK so word to the wise... if you're worried about AI replacing your networking job, start learning Infiniband and all associated technologies. Thank you Network Chuck and Juniper for this video!!!!!!

[@n19ence]

Fantastic cutting edge content Chuck...

[@OmarJuvera]

12:26 Chuck, this is precisely my beef with Ethernet. It did not "think" about the future, until a contender arrived! If Ethernet could do improvements to our current networking tech, why it did not? Precisely because it did not care...until it saw it's survival on the line. We could and should have a better network than what we currently use now, not just for AI. It is better to have both so they push each other to advance

[@filthyfrankblack4067]

True thats how Adobe has opperated for decades.

[@DJTimeLock]

That doesn't really make sense though? There's literally nothing to gain from "holding back progress". What is "it" in this context? Companies implementing network protocols? The people developing them? They all gain from having faster and more reliable products than the copetition.

[@pikpockett]

@@DJTimeLockThats true, however it seems that the standard itself had nothing to compete against, hence the lack of new ethernet protocols in the past decades

[@DJTimeLock]

@@pikpockett necessity creates innovation. That's very true. But to me it seems like it wasn't a necessity until the AI boom happend. As much as his analogy about a ferrari on blocked highway it goes both ways. What's the point in creating a 12 lane highway if 3 lanes are sufficient.

[@juz882010]

FC is just better especially for storage or RDMA

[@davidjv9800]

Yes, more data centers networking and AI please

[@maverickjones7087]

This dude makes me want to learn

[@oculuscat]

Based on the models I train at home: For RTX 3090/4090 cards, you'll hit VRAM limits before bandwidth limits for the size models you're working on. 10Gbps is more than enough. You don't need anything exotic. Also it's fine to put your GPUs in multiple cheaper computers; you don't need a single super expensive motherboard/CPU.

[@joseleperez8742]

AI training is usually done over A100 or H100 and you sure will hit the bandwith issue. A mixture of experts of 8x220B parameters models is not your everyday at home. Training requires communicating the gradient and optimizer which is about 3 times the model. This is several GBs at each iteration, while the GPUs have the model sharded and they just need to execute one layer per iteration which is way faster.

[@oculuscat]

@@joseleperez8742 Not sure if I care how you think its "usually" done. I see lots of papers presenting cutting edge work trained on 3090/4090 cards, sometimes just one. For home labs it makes no sense to purchase an A100 or H100 for many reasons.

[@juz882010]

It's more about time, the bigger the model the better it is. The faster you can build the model the better. We are talking about millions of hours of GPU compute.

[@tfpohistory]

your old music "all you wanna do" was pretty awesome. whynt you using in new videos?

[@DrakenBlackknight]

I have no problems with A"i" (really just the same machine learning we've had forced upon us for 10 years) being slowed to a crawl.

[@Gordy-io8sb]

Same. Infact, I am in *full support* of dismantling companies like Stability, Wombo, and Open. I have no problem if RatGPT is shut down, along with all of the other scummy A.I. applications.

[@TWARDOWSKY.]

I think the native speed of Chuck videos are -20% of "normal" speed, unless it's a matter of Check coffee....Check what's going on ;)?

[@pancake_king1106]

We're too far lost in the Ethernet sauce, to all switch now

[@v01dd]

This is a great video. Its been a while since i have felt excited about networking and technology. but this video has me super interested about AI and networking. this is the first time I have seen network specific design for AI.

[@Frog_Scooper]

I'm not sure if I should be scared or excited

[@pauljohnson2023]

really good content again Chuck. I might have an easier time picking between the two if I understood how firewalls and QoS work in the Infinaband space. Or do they have an alternate technology?

[@BadMoodGaming]

Awesome video chuck

[@jonathanrhodes6180]

2:50 Before we air cooled data centers, we liquid cooled them.

[@juz882010]

when? show us pictures lol talking shit

[@tsionwilliamson1385]

Based on your description of how infiniband works it seems like there being an internet connected device which will likely be running a full operating system controlling direct memory access between two computers could have some potential security implications. For instance hypothetically if a malicious actor were to gain access to a switch handling the connection between the two servers it could be possible to inject malicious instructions directly into the receiving servers memory allowing an attacker to potentially gain some level of control over the server having its RAM written to.

[@SimonWeider]

The lowkey roast at 6:26 😂

[@promiseenyinnaya6346]

Since we are exchanging data directly btw the memories of the two servers without going through the network stacks or layers, it also means that a large bandwidth will be required since there are no data formatting techniques such as compression and encoding.

[@WeylandLabs]

You're forgetting one crucial thing, sustainable power consumption at scale. This is all nice and everything, but until we can find better innovations in sustainable energy management systems to scale, it doesn't matter what type of software or hardware you have.

[@i_Kruti]

Thank You for helping us to know more details of this new networking technology.....!!! and by the way I support both..!!

[@TureIMasterEquality]

The "Animaniacs" part is a classic...😅😂 oh and yeah the AI slow down, good job Juniper..

[@Neckername1]

You gotta really break this problem down. It's specific parts of the networking that are standing in the way of having better bandwidth. There are also things that are more important at times to allow hardware and software to be able to effectively complete their operations efficiently. Latency is one factor that I have seen in software that can cause some software to behave different than it was intended to. Latency also drives up software costs because measures to contend with the varying levels of latency need to be put in place. I think we should be going hard into fully optical communications circuits. I mean, that everything is done via photons until the very last second. Decreasing lacency and allowing the bandiwdth to be even less constrained to the fabric/medium it was transmitted through.

[@chagle595]

this mutha trucka ... not ONLY is he using my name better than I could ever hope, but THEN you add on that MAGNIFICENT beard! ... THIS SHALL NOT STAND! (no really I'm a network engineer named Chuck) great information, my dude! For the record, Team Ethernet

[@redaxxx]

you're talking about infiniband and rdma being new.. they're ooold :D For instance Isilon used Infiniband for their backend network for over 10 years, they even have migrated away from it to ethernet already

[@Vatharian]

There are numerous initiatives that aim to reduce these issues. One of them is CXL interconnect, which takes RDMA concept like a step further and throws networking to top of the rack only. Since CXL allows P2P rootless communication, you can eliminate the "server" part from the server, and leave the resources only - RAM, accelerators, GPUs, storage, RAM (yes, as an actual device - there are already CXL connected RAM carrier boards) - and if you want - an actual CPU. This enables for example RAM sharing across whole CXL domain, and any network capacity added to the system benefits all of the devices equally, with basically fixed latency. You could even export and virtualize whole parts of CXL tree over normal network to group systems with identical dataset or mix and match for power envelope control. Of course it's not panacea - it's restricted to rack scale only solution, so you need same infrastructure anyway outside of it - just less of it, it's power hungry (current CXL switches eat up kilowatts of power), it's insanely expensive, and requires rewriting and validating most of current workload managers, which comes with its own challenges, kind of doesn't fit in normal systems, the hardware is quite fragile physically, and it's not something you can build over time - you need to set it all up in one go, which kind of raises difficulty of adopting it and raises entry bar for development, not to mention requires expert knowledge form much wider set of topics from the developers than otherwise. We will most likely never see much from this in "normal" datacenter environment for many years, as this tech is heavily skewed toward supercomputer systems, but the innovation is always welcome.

[@raphaelfranzen9623]

At this point , Ethernet is a known quantity. I see no need to reinvent the wheel .

[@patrickchan2503]

Hello Chuck, I appreciate all the stuff you do on KZbin. Maybe be a bit go easy on the editing, zooming in and out, etc, because networking isn't an easy topic and I'm a little overloaded. But you don't ave to change anything on my account 🙂 Thanks.

[@PotaytoDestroyer]

Infiniband seems like the best thing to learn right now for a network engineer.... where can I learn this sorcery....

[@Abduselam.m]

Wow really amazing topic Thanks so much Networkchuck

[@SirinAykut-yt9rh]

Just won on 4RA today, bro. Feels great! 🤑💪

[@seppo9455]

As someone who has extensive knowledge in installation/testing/troubleshooting network equipment/ethernet etc. I welcome the endless work muahahahaha

[@XtecherPY]

1:53 was crazy

[@michaeldancurtis]

2:10 bro InfiniBand has been around longer than PCI-express 3:10 again, this issue happened in 2003, hell all of 2004 and 5 was nearly fully dedicated to finding and powering datacenters 3:30 watercooling in a datacenter is also not new 5:55 RDMA has been around since 2013 ish and RoCE is like 2015 11:45 IPU and IDPF is gonna blow your mind

[@trevornelson1194]

Ethernet for sure. Can imagine changing the for on global scale🤯

[@omarsplaytimechannel]

THANK YOU FOR POSTING CHUCK!!!!!!

[@tunisiano187]

TCP/IP it is for me, as you said, that is well known, and easy to find help. But that will need a separated network to secure it.

[@muhammedhusseinkarimi1002]

RDMA is also used in some Clustering stuff for databases or Storage systems

[@DevanjanMaiti]

The industry is already moving back to ethernet mainly due to the need of L3 routed networks in isolated environments.

[@JuanDeLaRosaTV]

Are you suggesting that a career field for infini-band technicians is coming soon?

[@bennywenny_]

i’m really not sure on how to get started in IT out of highschool? do i go to college? get certs? i’m hoping to get into cybersecurity. any tips would be awesome

[@snail1833]

hey chuck been a sub for a while always really like what you're doing, you and many others are also inspiring people like me to also maybe go down the same road as what you guys are doing work for myself and share knowledge with people that can make use of your skills and passion instead of always fighting with cooks at the normal workplace.. thanks again.

[@edddieee-main-geee-]

If you look at what we've done with ipv4 to avoid ipv6, I'd bet Ethernet wins.

[@imperativeemarketing5546]

I'm not into Networking, but it seems to me that because it's lossless RDMA has a major advantage. You have to do all kinds of engineering to make Ethernet work at the same speed. It just seems to me that RDMA will free up engineers and hardware to do what needs to be done as opposed to making what doesn't work correctly in the first place to work correctly. It may take while, but I think that eventually the RDMA type protocol will win out.

[@roberto3815]

I work with Juniper on the daily think they are great but we use fiber not ethernet and can get 100 gigabit and that is just the communication to each other. The company I work for is a fiber network ISP we have no ethernet in the network outside of some old parts of the network we have not upgraded yet. But you are right about Juniper sense we switched to them from mikrotik.

[@realWorsin]

They did a study in 2013. They asked 2500ish AI researchers when AI would take over ever single job. They estimated 2045. Recently they did the same study. The results were pushed back to 2120ish. This is due to hardware limitations. They estimate 25% of the entire power grid will be AI alone. This issue is good guys, we need it to slow down and perhaps hardware limitations will be our saving grace.

[@georgelza]

love the new capabilities being added to the entire ethernet stack, but for HPC IB in my view just that more mature/proven. it's been the core of Oracle's Engineered system solutions since 2008, running on Mellanox switches. for general DC networks, sure ethernet, for something as specialised like HPC/AI stacks, I think IB still the leader.

[@joshuamercer854]

Very informative!

[@Comedykingdom69420]

Great thumbnail

It has some "I have no mouth and I must scream" vibes :)

[@GadgetReviewVideos]

Juniper has come a long way since that started. I’m for ETH, it’s not like we hit a limit with Ethernet that we can’t vet come like IPv4 so why reinvent the wheel from scratch.

[@Halfan_Hossen_Habib]

Amazing broo 😍😍

[@illustrio7077]

Hey Chuck, wouldn't RDMA require the data to be transmitted in an unencrypted form? Since the CPU is not processing it, I don't think any sort of encryption standards would be plausible for such networking, and hence it leaves the whole network more vulnerable than ever before.

[@1BadMother]

#1 This is usually local traffic. In the example Chuck mentioned most scenarios only involved switching and not a lot of routing. #2 You can very easily do line speed encryption/decryption in the NIC.

[@juz882010]

its converted to FC or Infiniband. Depending on if you NVMoF or RDMA infiniband.

[@juz882010]

@@1BadMother why would you need layer 3 in this scenario? please tell me you dont setup your local network like this?

[@MrMusiciluv17]

Insightful video. Thanks!

[@zeeby4682]

Thanks for that absolutely cursed AI generated preview.

[@krotson6767]

Why don't they use fiber optics everywhere? I guess they can afford it? If they increased the version of their IT system, there would be no problem. One fiber optic cable not enough? You have 10 more. And this is how you build highways between servers. :-)

[@rndmlogin]

Love your chan. Your videos are fun to watch and you are a relatable host. Thanks for all the great content! Infiniband is not new. Ive had infiniband in one of my production clusters for neigh on 20yrs! Infiniband doesn't have to be expensive. I picked up 2xQlogic 18port QDR switches for $15 on ebay(still dont know how it was possible) I was able to pick up cards/cables for 4 servers for in my home lab rack for under $200 . I run subnet manager on a couple hosts since these switches cant run it.

[@ksmit1030]

I’m team infiniband for these reasons, Ethernet is getting hard to get up to speed, I love Ethernet but eventually we are going to need speed and reliability that Ethernet cannot keep up with without major complexity added. The improvements mentioned in the video are great but isn’t that a lot more complex then just figuring out infiniband and future proofing it then having to upgrade every few years? Anyways, I had another reason but I forgot it sooooooooo. Just to be clear, I love Ethernet and want it to work well but it just isn’t fast enough and is getting bottlenecked.

[@juz882010]

We already solved the issue with FC. It doesnt drop packets but people too dumb to understand zoning.

[@vadymkvasha4556]

seems you're correct, and that isn't only in terms of community and support. What about scalability? try to make global InfiniBand network, seems it will be developed as local solutions for the datacenters

[@最も愚かな]

I really like the thumbnail

[@iMentorCloud]

well AI is already being used in networking for Automated Switching | Intelligent Routing| Quality of Service (QoS) Optimization | Self-Healing Networks | Anomaly Detection etc etc ;) good video

[@skeginaldp1533]

How do you stay in shape with all the things you do!

[@lspcie]

Interesting video but very high level. Regarding the high power consumption/heat dissipation point on GPUs a topic I’d like to see a well researched video about is the possible avenues to improve code efficiency where at a given load an LLM could have more productivity, as that would counteract the FLOPs/time needed and hence reduce power per output on the AI side

[@lidoreliya-9500]

when 800GB Will be available for purchase ?

[@Nate-wn1lk]

I love u chuck my inspiration ❤

[@TheFirmest]

When do you think graphene will come into the picture when it comes to building network equipment with it?

[@mounikapallaprolu6043]

Today is my first day of starting my career in cyber security... But still stuck in which laptop to choose in budget... Can you please suggest.....will be a big help for me....... please