Solving Linux DevOps Issues with OVER 1000 CORES!

[@fracturedlife1393]

Way over my head as usual but Wendell's enthusiasm always wins out.

[@shinokami007]

[@tanmaypanadi1414]

🤣 same here.

[@rax313]

Same

[@gg-gn3re]

video is pretty basic stuff so you should feel bad

[@MrViki60]

@@gg-gn3re ok codehead

[@JeffGeerling]

1:24 a man after my own heart.

[@NixiePixel]

"We're gonna talk a little about Linux" Me: 🐧🐧👌👌

[@emperorSbraz]

hello there

[@hongtanke]

So what you're saying is "Talk Wendell to me"

[@browaruspierogus2182]

Computer Science - a science of solving problems that would never exist if there was no computers

[@prostaknaivnyj5976]

so tru

[@Encysted]

Social Sciences - multiple sciences of solving problems that would never exist if we could just stop being in societies for once

[@guxtavo14]

"Bench-marking the Linux kernel compilation with kbench and kbenchrate" is the appropriate title for this video in my mind, but thank you for sharing

[@plapbandit]

_Kilocores as a unit of measurement_ What a time to be alive

[@SeekingTheLoveThatGodMeans7648]

This would be the equivalent of the proverbial old computer that fills a warehouse with dozens and dozens of CPU cabinets, but in one box smaller than a refrigerator.

[@SeekingTheLoveThatGodMeans7648]

And yet, the primary use of this stuff might not even be software development coolness, but scientific and business applications. IBM, for example, has 50,000+ MIPS business computers. Build the computers at a decent price and the applications will come.

[@bassstorm89]

1:00 "low hum" :D I had to genuinely laugh haha. on a real note: Wear ear protection, whenever you go into/near a server rack. High pitches can get really loud really fast

[@nekomakhea9440]

"You probably aren't going to convince your manager to let you re-architect a project for faster compile time" That's why you don't ask permission, you just do it on a separate branch and don't tell anyone about it until it's ready to rebase into master. I've done this on real production projects, during the downtime between release cycles where everyone had nothing to do but browse reddit for 3 weeks while managers and bureaucrats argue about arcane policy things for the next cycle, because I was bored. Surprisingly enough, nobody was mad that one day the CI pipeline was suddenly 60% faster and produced 70% smaller executables.

[@josehugoelsas8699]

"Better ask forgiveness than permission" someone told me one. Your story pretty much sums it up, and I have also lived through situations like this. If you simply show up with results, most people simply prefer not to say anything (and smile) or try to present the sudden improvement as their own.

[@d00dEEE]

How does scaling_governor set to schedutil fare against performance? Better than ondemand?

[@supercheetah0]

Building a machine for GKH? Nice. Did you ask him about what he thinks helps with speeding up kernel compilation? Maybe playing around with gcc options might change things, like -O0 so that it just skips any optimizations in favor of spitting out binary as fast as possible (and then it would be interesting to see how that affects the performance of the final compiled kernel with those optimizations). Also, what about using ccache? Some of those common header files surely could benefit from that?

[@SeekingTheLoveThatGodMeans7648]

I'd guess that each core is sicced at a different compilation process, all going at the same time. That takes care of the CPU needed. I/O (or RAM contention) could still be an issue.

[@Hoztag3]

BRO i really wish i knew what u were talking about, i just think its dope listening to u talk about tech, much love G

[@CrazyGuitarTV]

Great Video. I know I am asking for a lot here. But could you also cover the ongoing rage in Linus new video about AMD CPUs being locked down by Lenovo, Dell etc? I hate that everybody is like "I am never buying from Lenovo again" Sure, there should be an option to activate it or not when buying. But there's a reason why some company's demand that feature and it was not up to the OEM's to use PSB or not. People forget that those systems are for enterprise users and not for your typical office or gaming room. I know ServeTheHome's did a really good video about it like a year ago, but I'd really like to hear your opinion about it. Have a nice day and thanks for the great content.

[@husamabou-shaar9740]

When you run multiple compile jobs, does the bench tool force scheduling on a specific CPU/CCD and its corresponding NUMA zone? If not, I think it would be interesting to do it manually, one method comes to mind, maybe try to sandbox the benchmark in a container, run multiple jobs/containers in parallel while forcing each container to live on separate cores and NUMA zone.

[@pyroromancer]

Wendel has awoken

[@asdf51501]

Wendell: “People a lot smarter than I am…” Me: Realizing Wendell is a lot smarter than I am. 🙁😂

[@Encysted]

Were these benchmarks run from a kernel that was custom compiled with a fully custom compiled toolchain?

[@AdmV0rl0n]

Wendell in full mad science mode :)

[@maltoNitho]

This is the nerd stuff I come here for!! I will however need to watch it again to catch everything. Can we talk about Wendell nailing that outro though?! 👏👏

[@mattriley6283]

Some very interesting content here, You do make a good point about cpu cache and clock speeds... I'v ran a dual HT hex core system in the past and i ran folding at home on it having the dual sockets and two lots of cache did offer certain advantages performance wise when doing heavier jobs and many at once. Looking forward to seeing how all this develops, are there any existing Linux Distros or kernels you would recommend for workloads that use lots of cores and high usage for long hours?

[@andrewr7820]

How about Watts per Kernel? Efficiency is becoming the limiting factor for HPC implementations, as there is generally an upper bound to how much capacity a power company {can | is willing to} deliver to a single location. I've seen 6-8 MW quoted frequently, though I'm not an expert. The Top 500 list has a related Green 500 that explores this axis.

[@RiffyDevine]

Weird question but that system you had in a normal case running two cpus, is it that quiet or did you just not use audio? I am at that tipping point of building another server in a case or just buying used server gear.

[@AmnesiaPhotography]

It’d be interesting to dig into the make file too and see if there were any optimisations that could be made

[@hwetherell6250]

Wendell. You need to include a link in the description to your linux channel. You mentioned it twice in this video, but unless I'm missing it, it isn't on youtube. If the linux channel is on your level1tech forum, then it isn't clear I need to look there either. Thanks for the video though!

[@Delijohn]

Level1Linux

[@hwetherell6250]

@@Delijohn thanks!

[@tanmaypanadi1414]

Level1Linux kzbin.info

[@bassplaya69er]

just go to the home page of a channel, and then to the channels tab to see their other channels

[@johnxu6751]

I wonder how much difference would switching to mold (linker) make.

[@MrTubeuser12]

" the server chooses the admin " was that a harry potter reference ? LOL (the wand chooses the wizard)

[@user-rc9jf8ng2k]

Sometimes I just watch these videos even though I don't have a clue what he's talking about.

[@cameronkeenan7766]

So I'm not understanding even half of this but I know what compiling is so I have a question. Would the actual use case for something like this be to create a whole bunch of "install discs" to sell like back in the Windows XP days?

[@briantoga2626]

What happens if you cluster them? I wonder what the results would be. What network speed would you need?

[@wpyoga]

Yeah that would be interesting! Not sure if different compiler versions will produce incompatible code, or if the same compiler running on different CPUs will produce incompatible code, I'm guessing there will be some kind of configuration option to avoid that.

[@LtdJorge]

@@wpyoga if you are using distcc, every compiler will use the same target, unless you pass -march=native, which will compile in each CPU for that specific architecture.

[@wpyoga]

@@LtdJorge I see, today I learned something, thanks!

[@Dr_Nick_]

Man this is the slowest popcorn popper I've ever heard of.

[@progressivenetwork3584]

Can you post the kernel building instructions on how to set it up too

[@Level1Techs]

Kcbench, it's the forum thread link in the description

[@progressivenetwork3584]

@@Level1Techs which instructions are you following.. what dir to put a repo in etc

[@Level1Techs]

Kcbench does it all for you. You can do the command examples given on the forum and have a like for like test assuming the compiler is the same version too

[@joshxwho]

This is probably a super niche video, but it's my niche and I love it. I love this video.

[@sbrazenor2]

Any way to use the GPUs as some kind of process accelerator? 🤔

[@leviathanpriim3951]

wow, not what I expected but great info thanks Wendell

[@ask_carbon]

Anyone catch which kernel version was Manjaro at during all the tests?

[@1slotmech]

Wendell, this whole video was way over my head, like me standing at the bottom of the Marianas trench over my head. But hey the wizards got to be wizards. Me, I'm just going to throw Mageia on my old FX 8120 box and go watch some Netflix to try it out. 😜

[@sharkbytefpv4326]

Wendel, you are the nerd of my dreams. Keep up the good work.

[@Cooper3312000]

This case and motherboard are a fantastic match. I'm just using a pair of 7302's with only 256GB of ram but it's still an amazing setup.

[@TerminalWorld]

If you could link to 'Level 1 Linux channel' that would be great.

[@tanmaypanadi1414]

kzbin.info

[@AegisHyperon]

How many DevOps per second is it?

[@ArcadiyIvanov]

Have you tried CCache? :)

[@TheBackyardChemist]

a kilocore is a very modest allocation on most of the modern HPC clusters on Top500 :P

[@AJMansfield1]

You should try setting up a tool like `distcc` to see if you can use all 1000 cores at once across the entire cluster, all on the same job.

[@memberman]

Super cool to see these beast CPUs in action.

[@keyboard_g]

You can now compile the Linux kernel before I can figure out how to quit Vim.

[@wpyoga]

Jokes aside, Vim is actually quite forgiving. If you press Ctrl-C, it will tell you how to quit.

[@JamesSmith-sw3nk]

I've been watching for 5 minutes.. When does Wendell start cpu mining Monero? 😁

[@johnmijo]

Very nice Wendell :) Now then, somebody needs to a create an alias using the -thicc switch to replace the -m switch, ha :p

[@keremozturk4134]

Me, broke student, living off my quad-core: Hmmm, yeah, this video is for me 🤣 Thanks for the tech madness I love to see 👍

[@seylaw]

Used Xeons might help you to overcome the 4-core era on a budget.

[@deth3021]

Most devs work on 4-6 cores.

[@seylaw]

@@deth3021 There is no need to limit yourself, you can get a 18-core Haswell-EP Xeon for around 120 USD. That makes compile jobs faster and helps to make devs more productive. ;)

[@deth3021]

@@seylaw most devs use company provided kit. Also it isnt a limit. World builds are for build servers. Incremental builds is enough for what most people do.

[@seylaw]

@@deth3021 Fair enough, but as the OP seems to be a student with little money and complains about few cores, I just wanted to point him to some budget options to overcome this limit. As long as you are willing to buy used hardware there are options which he might not have thought about before.

[@FrancoCastro]

htop in the background that is something I can respect.

[@Vegemeister1]

How about schedutil vs ondemand?

[@kjlovescoffee]

Damn, only ten kernels an hour? I remember when compiling the linux kernel *once* took more than an hour. Much more than an hour.

[@Bob-of-Zoid]

That's what I always wanted ! 1,000 processor cores!!! My Penguins are dancing!!!

[@AlexSchendel]

As someone who's worked on OpenBMC... I would kill for a build in 2 minutes haha. I'd be very lucky to get a 2 minute build if I've literally changed a single package so just one module is re-compiled and then the filesystem is rebuilt and the image is zipped back up... If I have a clean build doing the several thousand build tasks, I'm lucky if the build finishes in under a half hour...

[@Level1Techs]

Think it's possible for me to get a generalized script or more info so I could build a benchmark around what you're doing?

[@AlexSchendel]

@@Level1Techs Ooh I'll be sure to try putting something together! I'd be very excited to see a video about building OpenBMC :)

[@OLDFLMAN]

You go Man !

[@Winnetou17]

I wonder what a young Dennis M Ritchie would say about this :)

[@radomane]

This is so futuristic you used unreleased, undeveloped kernel releases?

[@Level1Techs]

It is for gkh after all :)

[@Marc_Wolfe]

Your clickbait title ended up be more interesting than I assumed.

[@Ladsoftware]

arch base for devop??

[@deathdoor]

Honest question: why do you need to keep compiling the kernel in 20 seconds?

[@maxbartlett5002]

I think he's using it as a common measuring stick between systems. Also in the linux world even those who have never programmed may have built the kernel ( a while ago it was common to improve speed/performance/tweaking features ) . I suspect he's been building stuff for embedded systems and so this vid came about.

[@LtdJorge]

It's just a benchmark

[@gabest4]

Please no SI units. Next thing you know, they sold you a 100 core cpu instead of 128.

[@LtdJorge]

Next time, measure CPUs in cores per inch, or cores per pound.

[@ewenchan1239]

I confused. What's the "over 1000 core count" for? I thought that this video was going to be where you have parallelised the kernel compile over a HPC cluster. That would've been cool. Turns out that's NOT what this video was actually about.

[@alantuttphotography]

What kind of ROI does this type of benchmarking and tweaking lead to? How does all the time spent on this pay off in the end?

[@LA-MJ]

Us watching the vid helps some

[@tanmaypanadi1414]

save time and use that time budget to tinker more.

[@aghaeis]

This is a tech channel for other tech channels. I watch every video but usually understand about 5% of what Wendell says 😂

[@Im1CrazyCow]

World: We Cant get Linux Pass 20 Kernels? Wendell : Hold My Diet Orange Crush !!! 69 th Comment too 🤣👍💯🐊🌴🙏✝😎🤠Cow}:-o)

[@Theorica]

I'm thinking of changing my speedometer to Kernels per hour, now.

[@kortaffel]

Ehm, why not use DistCC? You can run that on all of your workstations.....

[@Level1Techs]

That was the 4 node cluster I mentioned :)

[@xurtis]

Here was I hoping to see an Esperanto

[@nirajbhagchandani1634]

Hi, I would really like to help u remotely if you allow me. I love Linux and would like to explore more on the Linux compilations and other kernel experiences.

[@enthusiasticpaunch]

That server sounds like pneumatic tooling...

[@jff400]

And here I am with my E3-1231v3 getting 1 kernels/hour .

[@mritunjaymusale]

Wendell needs to be stopped he's turning into a mad scientist

[@dustsucker4704]

Haha lol. 20 seconds for a Linux compile? My hello world in react can do that too.

[@b2bb]

gotta love them server fans... always reminds me of Linus and his banana--and odd trips to a data center as a child _engagement_ Edit: Okay, this is a disgusting amount of compute...

[@wagmi_dude]

Red eyes? Has Wendell bought bitcoin?

[@etherboy3540]

I remember back in the DX4 486 days it compiling the kernel took 2+ hours.

[@kamertonaudiophileplayer847]

It reminds me sound of a computer center where OS360 runs.

[@PupShepardRubberized]

Sooooo many cores :v Was expecting something different from the title :/

[@antonmuffin6214]

I was expecting a 1000 core system or something that combines multiple systems together, I also dont know how you edited your comment 3 days ago if it says this video was uploaded a minute ago

[@Level1Techs]

We cut the distcc part for time. Didn't help much and setup was a pain

[@chrisrib05]

Meanwhile me with ~50 cores spread on almost 10 machines _its fine_

[@adhdengineer]

1024 cores dammit.

[@celestialbeing4767]

Rustlang

[@level80888]

umph

[@deusxyz]

Numbers go brrr

[@dakata2416]

Gentoo users be like

[@asdf51501]

Last time I ran Gentoo was back in ‘04 or so. I wonder how it is these days? (My dumb ass should just fire it up in a VM, give it 24 cores or so, and find out.) 😃

[@dakata2416]

@@asdf51501 It's pretty good

[@scottmckenzie4227]

DevOps issues? This video has nothing to do with DevOps.

[@deinemamainhd]

Kilocore? Translate to "freedom units" pls 🤣

[@lamikal2515]

Laughably pityful... I would only consider 9001 cores as enough to solve those issues.

[@Nickscrazylips]

make - j 10000 or gtfo

[@MikeG-js1jt]

How is it that you're LADEN with all this awesome hardware??...... are you paying American money for all that??..... You're like that Duck McBucks throwing all his gold coins up in the air, but your in the garage digging through crates of cpu's....... a constant influx of BS!!

[@rahulprasad2318]

Touch grass bro

[@TechySpeaking]

First