Speed Up Those Emerge Times in Gentoo

[@adarshsingh764]

4:35 from the man page for nice "Niceness values range from -20 (most favorable to the process) to 19 (least favorable to the process)." so if the nice value is 19, it will be the lowest possible priority, and will not kill other apps.

[@killistan]

I set mine to 15 so I can still use my system while compiling. It doesn't seem to affect emerge times in any tangible way... Just prevents most of the buffer underruns with alsa. (I used to not even have to nice anything :-\ )

[@theusernameyoudidntexpect6511]

I think there's some missing information here about how MAKEOPTS and emerge's --jobs arguments play together. The way you have it currently configured (per this video), you would be using up to 64 threads when building packages simultaneously - 8 emerge jobs, each with 8 make threads. This is rather sub-optimal given most domestic CPU's are 4 to 16 threads, though it's mitigated somewhat by using the load-average to limit new threads. In addition to this, you don't consider memory requirements. Each make thread can consume a considerable amount of memory. It's not much for most smaller packages, but you will commonly see 2GB or more RAM used per make thread. That's 128GB RAM just for building, if fully loaded using the 8x8 scheme you describe. Having said that, portage tuning is mostly a trial-and-error experience, and different settings will work for different people with different use-cases and different hardware.

[@theusernameyoudidntexpect6511]

@@leonlin5274 The system should generally relinquish file cache when an active process needs it; but yeah you'll see more problems with bigger packages if you use more aggressive options.

[@user-ke6gn8pg3u]

It wouldn't reach 64 threads simultaneously due to the current loadavg + loadavg of dependency calculations; emerge child process initialization; downloading / reading from disk, hashing, and unpacking the distfiles; as well as the single-threaded configure scripts that usually get executed before parallel make jobs can be run. I could see maybe around 18 threads running when it first starts but going down to 8 - 10 afterwards (less than $(nproc) threads if there was a huge stall caused by I/O like swapping since that counts for loadavg). If you were using LTO system-wide you'd certainly shoot yourself in the foot with memory usage if they were all linking at the same time.

[@theusernameyoudidntexpect6511]

​@@user-ke6gn8pg3u It's a theoretical maximum, sure, and I did note the loadavg would help limit it, but just relying on loadavg isn't the best way to manage it. By that logic, why not set it to 64x64 jobs? Your loadavg will stop it getting too loaded, right? Besides, it's not having an excessively large number of threads that's the direct problem here, it's how much memory they need. Some people have systems that could support it, others rely (far too heavily) on swap, but many will be using systems in the

[@petrusion2827]

(sorry to reply to an older comment, but...) So how do you achieve being able to compile multiple smaller packages at the same time without the risk of starting more compile threads than your threadcount (or maybe let's say start maximum of twice the amount of your threadcount if you have 4GB of ram per thread to spare), and at the same time be able to give the full amount of threads to one large package? Do you not use MAKEOPTS and just use EMERGE_DEFAULT_OPTS instead?

[@theusernameyoudidntexpect6511]

@@petrusion2827 There is no way to automatically give more threads to bigger builds. You have to either find a balance between "not too many threads for small packages" and "not too few threads for big packages", or manually tweak your MAKEOPTS per-package. On my 12-thread 32GB system, I tend to stick with MAKEOPTS="-j4 -l11" EMERGE_DEFAULT_OPTS="--jobs 3 --load-average 8" - build up to three packages simultaneously with up to four threads each, but don't spawn a new thread unless load is < 11, and don't spawn a new package build unless load is < 8. This lets most things build quickly enough while giving bigger packages some oomph, without totally soaking the system and making it unusable.

[@austinramsay]

I didn't even realize the EMERGE_DEFAULT_OPTS variable before. Just set that up on my system and will try it out after reading the wiki page on it thanks! Another good thing to mention is setting up '/var/tmp/portage' as a tmpfs right? Uses more RAM, but if you have it then it is definitely faster and saves a lot of usage and writing to an SSD.

[@seancondon5572]

One of the best things to do with a gentoo install - at least on a Raspberry Pi - is setting up distcc. distcc is a MUST-HAVE if you wanna get stuff done FAST.

[@treahblade]

I know this comment is over a year old but if another person is seeing this be aware that you will need to fiddle with emerge quite a lot with distcc. I use it on my pi4 x 4 cluster all running gentoo and I have run into some strange behavior with some packages that just do not like to play nice with distcc.

[@seancondon5572]

@@treahblade as have I. In most cases, you just have to tell distcc to use at least one local core. Didn't take me long to figure that out.

[@stephenreaves3205]

Also, portage 3 is now stable and reports up to 50-60% speed improvement for dependency calculation (although I haven't tested it specifically)

[@marioschroers7318]

Excellent content on this channel. I decided to transition from Arch to Gentoo, though it'll take some time to get prepared. I'll probably need to run through a couple of installs on VMs before taking it to real hardware. Thanks for sharing your knowledge!

[@Jrv001]

You've got the juice! Thank you - very informative.

[@sage5578]

That thumbnail is fire.

[@user-ke6gn8pg3u]

Load average includes the time processes are waiting for disk I/O - which is likely to be a factor if you have many compilations running or your system is swapping memory. So, it can actually lead to a situation where you have less threads running than the number of threads on your CPU since it takes 60 seconds for the load average to catch up to the current load. Also, emerge doesn't invoke the `nice` core utility. It does it internally eventually executing the setpriority() system call. You say not to set the nice value too high. Nice values only have meaning in the context of other processes, so under low no system load it makes no difference. It sounds like you have the assumption emerge is forking off processes with lower priorities than itself - which is false. POSIX processes all inherit nice values from their parent. The only time nice values are ever set during an emerge is when emerge is first run, so all of its child processes: the builds - have the same nice value as emerge. If your intent is to set the lowest priority possible, you can use the chrt utility to set the scheduling policy to SCHED_IDLE - which is lower than all nice values with default scheduling policy. If you set idle scheduling policy, setting a nice value is not needed unless you are running other SCHED_IDLE tasks at different priorities. If you don't want it to have the lowest priority and you also don't like the semantics of process nice values, you can play around with autogroups (see `man 7 sched` for more details).

[@yoshi314]

why would niceness kill other processes? this is not oom_adj in a memory leak scenario? this may slow them down, sure.

[@jpgaminf7834]

priority

[@BandanazX]

You can also add "parallel-fetch" in the FEATURES which might help a little

[@zolaarczakle]

In the Gentoo Wiki, MAKEOPTS is set to the number of CPU (wiki.gentoo.org/wiki/MAKEOPTS) but in the Handbook, they say: "A good choice is the number of CPUs (or CPU cores) in the system plus one, but this guideline isn't always perfect. " (wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage). Is one way better than the other?

[@theusernameyoudidntexpect6511]

The "N+1" scheme is mostly a holdover from older times where CPU's getting I/O bound would cause unnecessary delays. You could essentially overload the CPU since it was busy waiting for the disk anyway. Having said that, though, fine-tuning portage depends more on your hardware and use case. Try a few variations and see what works best for you.

[@1337x1337x]

ccache, CHOST, and CPU_FLAGS_X86 (cpuid2cpuflags)

[@Flash136]

@Caio Novais Mind elaborating on how ccache is placebo? I've tried backporting features to older kernel releases before, and I was using ccache that time as my machine is rather old. I definitely saw an improvement compiling with ccache vs without it. 1hr builds turned to about 15 mins. And my CPU wasn't reaching 80+ temps anymore. Of course, you're not going to get the same benefits if you're *always* building for vastly different codebases. But for multiple builds on one machine, I find it rather useful.

[@user-ke6gn8pg3u]

@@Flash136 it can be slower if you use a HDD and the system has a high I/O load already, or the time required to invoke ccache, hash the input files + flags, look up if a file is in the cache, load a cached file from disk, and decompress it is slower than the time it takes to execute the compiler directly.

[@DimiEG]

Now Gentoo has gentoo-kernel-bin which improves speed of Gentoo installation. They use not optimized kernel config from Red Hat. Sure, it loads much unnecessary things, but allow to lazy users forget about kernel configuration.

[@DenShustrik]

Спасибо, отличное видео.

[@davidyoder5890]

Maybe you could make your term font a little larger for videos like this. It's pretty much impossible to see on a phone lol

[@vedantganesh6923]

Hey, do you mind posting your dots/programs? I'd love to try a WM on my gentoo box.

[@wayneferguson14]

What do you think about slackware

[@zyansheep]

Video on Nixos?

[@DimiEG]

Great video. Helped a lot. Would you show how to run podman (docker) on Gentoo? Tnx.

[@f23anone82]

Could you provide a link to your config files?

[@Lowacy]

confusing that higher numbers are lower priorities

[@juzujuzu4555]

The more nice you are, the more you wait others and allow them to use CPU :)

[@Lowacy]

@@juzujuzu4555 ah yeah this makes sense :)

[@skittlesvampir8400]

What colorscheme are you using?

[@mercuriete]

That's incorrect niceness should be 19. With 19 not only is low priority but is batch scheduling. If you use 19 you can use OBS at the same time you are compiling chromium and there wont be any frame drops. With 1 the process will compete with the others in your system. PS: you forgot PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}" You can search for that configuration here: dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html

[@ITBoys4]

what about ccache?

[@isamuranable]

Missed opportunity to name the video rev up those emerge times in gentoo. (Rev up those friers)

[@laughingvampire7555]

nice of 19 is the least priority dude, if you set it it wont kill anything.

[@LukasTheWizard]

gentoo go brrr

[@givlupi2686]

What do I do when I don't have the same lines you do? My USE and MAKEOPTS lines aren't even there.

[@IllicitPrism]

Add them yourself, it’s safe to do that

[@mega_gamer93]

make them

[@givlupi2686]

@@mega_gamer93 Thanks but very late. I figured it out but then gave up because I kept getting kernel panic. Now I use PopOS.

[@censoredterminalautism4073]

Install Exherbo.

[@seancondon5572]

Regarding the first sentence... what OTHER source-based linux distros are there? I mean, really, every distro that ISN'T BASED ON GENTOO... is a binary distro. The only other "distro" that ISN'T a binary distro is LFS.

[@copper4eva]

Slackware is source based. I believe void is both source based and binary, haven't used void, so I can't personally expand upon how that works. There's probably others.

[@seancondon5572]

@raine OpenBSD? Hm. Then I shall have to give that a try next time I decide to run cross-distro tests on Wine.

[@____-gy5mq]

Freebsd's ports collection is what inspired Gentoo's portage.

[@mitchelvalentino1569]

KISS Linux

[@BijmanPL]

I would suggest trying Source Mage and Lunar (both share heritage to inactive project called Sorcerer) for different take on how to configure build flags, which is probably more grainy than Gentoo's Portage. Another interesting distro is Exherbo, which has similar USE flag management with config files like Gentoo, but uses Paludis - written in C/C++ with builtin additional tools - instead of Python based Portage. KISS Linux is good choice, if you want very minimal and simple distro. There are also GoboLinux, with different filesystem hierarchy, and ports based CRUX.

[@J3zu5]

i clicked on this video just to ask Who and why use gento gento for what exactly, daily driver - meme - flexing, srsly for what?

[@cranknlesdesires]

It's a fantastic learning resource as it gets you to think about what each thing in your system is doing and it also gives you a lot of control with great tools for handling your system and once you have built your system the way you want it, it's done. Nothing unexpected will happen. It's kind of like modding a game like skyrim, getting all the combat mods you want, a perk overhaul, textures, and enbs to make it look as pretty as your system can handle. Something about the experance so tailored is very satisfying.

[@J3zu5]

@@cranknlesdesires what about Arch ?

[@cranknlesdesires]

@@J3zu5 arch holds your hand more and limits you in certain areas, on Gentoo you can choose any init system you like whereas on arch you only have systemd unless your using a fork

[@J3zu5]

@@cranknlesdesires fck, when i installed Arch i thought i was cool, not anymore :(

[@Jorge-xf9gs]

@@J3zu5 Arch is fine. I'm switching to Gentoo because it gives me fine grained control over literally everything in my system. It's the only distro in which I can use all the libraries and utilities I want (Musl, Runit, LibreSSL, BusyBox, etc.). It can also be the fastest or the most secure Linux-based OS depending on how you configure it. The level of customization over everything and quasi maniatic control you can have in amazing.

[@egg5474]

Help I installed gentoo and when I typed help into konsole it did nothing???!?!?

[@godarayudhvir]

2544

[@copper4eva]

bye now Idk why, but that made me laugh.

[@WasabiWarlord75]

First

[@francescominnocci]

Nice

[@almasabdrazak5089]

Too hard , why not use Windows instead ?

[@barscanguvenkaya4706]

just get a macbook lol don't be so cheap

[@____-gy5mq]

Windows used to be easy. Not anymore.