NYLUG Presents: Lennart Poettering -on- Systemd in 2018

[@cthudo]

I like the direction this is going, especially if they manage to keep the computational overhead small, this is exactly the kind of per-app isolation a modern OS should have. Making this happen without bloat will be the ongoing challenge though, both on the systemd side and the application/configuration side...

[@csebastian3]

Fantastic. Lennart's work is amazing.

[@mcol3]

Starts at 2:55

[@nibblrrr7124]

3:58 ;P

[@capability-snob]

So good. I'm so glad they have replaced the glibc resolver as it was such a pain in the neck - ambient and synchronous.

[@caidengatlin2007]

i guess Im kinda randomly asking but does anyone know of a good place to watch new tv shows online ?

[@dominikjericho5749]

@Caiden Gatlin Lately I have been using Flixzone. Just google for it =)

[@alijeffery8518]

@Dominik Jericho yup, I've been watching on flixzone for months myself =)

[@caidengatlin2007]

@Dominik Jericho thanks, signed up and it seems like a nice service :) I appreciate it!

[@dominikjericho5749]

@Caiden Gatlin Happy to help =)

[@BAgodmode]

Finally to hear from the man himself

[@anitsh]

Coming from just watching other videos from Lennart. Loving em all...Cheers,

[@acagastya]

Could you consider releasing this video under a free-license like CC BY?

[@jindiggs]

Go Lennnnnnnnart !!! Wohoooo !!! I love you

[@vasylvavrychuk5374]

It would be great to have presentation linked around.

[@nicoladellino8124]

Nice work Lennart, i like systemd

[@RadioMartyT1B]

I think systemd is a good idea, but Lennart is a very abrasive and adversarial personality type. Sorry, someone has to say it. His attitude towards other devs is likely why there's so much friction towards systemd.

[@maxsnts]

@@RadioMartyT1B Hehe True but Linus is the same way (maybe worse) and almost everyone thinks he is God. I guess friction comes from people generally not wanting change... then blaming it on Lennart

[@Zmej420BlazeIt]

@@RadioMartyT1B I don't think Lennart is adversarial like Linus actually. I think he's ignorant and arrogant. He keeps reinventing things that UNIX systems solved nearly 30 years ago, and in a way that is basically proprietary. Here's just a single example: he talks about how systemd can create a temporary user for a service/unit. That's cool and I really like it actually, but he says systemd doesn't write to /etc/passwd . So that means the only interface is a binary interface with systemd. One of the most basic principles of Unix systems is that everything exists as a file. Memory is accessed not through arbitrary numerical positions, or application binary interfaces, but through a know file path. For example, /etc/fstab describes devices to mount at boot and /proc/mtab describes currently mounted files. If systemd respected the way that the system works, it would provide an interface accessible directly through the kernel, but instead it requires everyone and every application to use it's own interface.

[@xnoreq]

@@Zmej420BlazeIt _"I think he's ignorant and arrogant. "_ How ironic coming from a guy whose every single post contains utterly ignorant statements expressed in an arrogant manner. You again don't have a clue what you're talking about.: _"He keeps reinventing things that UNIX systems solved nearly 30 years ago, and in a way that is basically proprietary. "_ LOL, utterly ironic again because Poettering's software is free / open source software while UNIX was proprietary. _"That's cool and I really like it actually, but he says systemd doesn't write to /etc/passwd . So that means the only interface is a binary interface with systemd. "_ Again an utterly ignorant statement. First of all, /etc/passwd is not a list of all users available to the system. It's just one backend of many. No properly written Linux application that wants to query the whole user database just reads the passwd file. The actual user database can be queried with a glibc function which will also include NIS, LDAP, systemd or any other nss service. _"So that means the only interface is a binary interface with systemd."_ No, this has nothing to do with systemd. This functionality has been developed 20-30 years ago in most Unix-like operating systems. _"One of the most basic principles of Unix systems is that everything exists as a file. "_ Which Linux violates on many counts. Linux is not Unix, you know? Even something as trivial as the kernel's last output messages, essentially a log, is not available as a file. That is not a problem though. All you need is an application that acts as a source or a sink. Then have your script read or write from/to a pipe instead of a file. You already do it with a lot of things found on Linux systems ... such as dmesg, iptables-save/restore ... _" Memory is accessed not through arbitrary numerical positions, or application binary interfaces, but through a know file path."_ Wtf? Memory has always been accessed through addresses ("arbitrary" integers). Also, in Linux many APIs just take fds (again "arbitrary" integers) instead of file paths. Many operations in a modern Linux system would be much, much slower if everything was implemented as files specified through paths. _"If systemd respected the way that the system works, it would provide an interface accessible directly through the kernel"_ Again, the user database has nothing to do with systemd. You can query it the same way as you did 20 years ago. Systemd just adds another "source" if you will (using old standard APIs) ... which makes a whole lot of more sense for *transient* users than editing the local password database file (which again also would be slower). /etc/passwd is a mess anyway, because you shouldn't edit it directly with a text editor for obvious reasons. Hence all the user management tools that Linux distributions have shipped for decades...

[@Anonymouspock]

@@Zmej420BlazeIt lolwat. Have you ever heard of nsswitch or considered how LDAP works on a Linux system when you have hundreds of thousands of users? Hint: they're not in passwd.

[@RogerBarraud]

Does Anyone Really Know What Boot-Time It Is?...

[@alparslanmetehan9678]

I like runit btw

[@dipi71]

Nice talk - systemd managed to make me interested in booting fast again on my GNU+EFI+GRUB2+systemd+Qt+Gtk+Linucs rigs. :-) 8:00 Hope cgroup_disable=memory will still be possible on cgroupsv2, is this equivalent with »MemoryAccounting=on«? Nice to hear systemd uses the Berkeley Packet Filter (BPF), an exciting new playground (well, not really new, but still): service-level firewalls, sandboxing etc. 44:12 nice journalctl hacks right there 1:18:20 portablectl as a chroot substitute, sounds nice; resolved and DNSSEC verification, nice; good riddance to nscd and NetworkManager’s dhclient. Lennart’s answer to the last question made me smile. Cheers!

[@dipi71]

@Gary Tivey That's true for servers where the main aspect of boot minimization would be system hardening and security aspects (after all, what won't run won't cause vulnerabilities). On the other hand, I don't let my systems waste energy while I'm away. So frequent shutdowns and boots are just part of what I do, and minimizing boot times - and reducing startup complexity - have become one of my side hustles. A few weeks ago, my main rig managed to reboot in under 760ms, according to systemd-analyze (GRUB to KDE+networking).

[@formbi]

he seems like a nice person, I don't understand the hate

[@zaidgharaybeh8422]

His linux architectural positions are abhorrent. Don't care about what he "seems" like. Technology first.

[@Diggnuts]

The average IT hippy isn't that smart and can't abide change. That is the bulk of it.

[@zaidgharaybeh8422]

@@Diggnuts nah, it's just negative change

[@Diggnuts]

@@zaidgharaybeh8422 I'd argue that SystemD has been one of the most important and positive things to happen to Linux especially outside the mobile install base since Ubuntu started. If only graphics server or interface groups could get their shit together like SystemD did, we'd have a viable desktop ecosystem years ago. The old UNIX dogma is out of date and does not work for the average user. Without SystemD, *NIX based OS's on embedded would sport their own proprietary version of LauchD or whatever, or even have ditched Linux for BSD. If you still think SystemD is a negative change, you are inherently arguing that it is worst than Linux becoming obsolete completely.

[@zaidgharaybeh8422]

@@Diggnuts I think that systemD is good for short term growth of Linux and its user-base because it streamlines a lot of things, but in the long run, there's going to be trouble because of an overly complex, over-engineered system. This trouble may introduce itself in the form of compromised security, software inconsistency, confusing behavior, less user control, and bloat. These effects might happen in a long time from now, so you're right that it's arguable that going for systemd is the right decision. But don't pretend that linux is unix inspired anymore, and don't expect that a building can keep growing on a shaky foundation.

[@Canadian789119]

What happened to the way sound is done that I need to run a Server just to get sound on firefox?

[@outputresistance]

Well, you also need to run the X server to get graphics...

[@Canadian789119]

@@outputresistance Yeah, true. Good for multi user environments. I'm talking about mozilla making hard dependencies.

[@goolaguser3702]

@@Canadian789119 Well, patches exists to make Firefox support raw ALSA. Arch Linux's official Firefox package has that. Nothing wrong with Mozilla only writing for the dominant API. PulseAudio after all even works well on BSDs and Windows too and moves the dirty details from application code into server code. MacOS has a similar system called Core Audio.

[@overtheworl]

@@goolaguser3702 no surprise bc poettering _really_ loves to copy mac shit to linux

[@armynyus9123]

My Hero! systemd-nspawn >> Docker. Containers with systemd inside rock, fck single process containers.

[@2disbetter]

I like systemd, but I do wonder about his comments on swap files and them being on the way out. I do wonder if swap is on the way out, how will things like hibernation work? When of the biggest annoyances to me as I started using Linux was the absence of hibernation. You can suspend or shutdown. What if I want to maintain a session, but NOT run down my battery while it is not being used. This is why hibernation was even created. When does Linux seem to not use it?

[@steinbauge4591]

Portering, yes that Microsoft bloke

[@bigfootisjustreallyshy]

Lennart just wants Linux to be Apple. Systemd = Launchd, PulseAudio = CoreAudio.

[@YoloMonstaaa]

Yeah and there's nothing wrong with that, both technologies are quite good.

[@YoloMonstaaa]

@Finxert ­ Which of your values doesn't align with systemd or pulseaudio?

[@YoloMonstaaa]

@Finxert ­ Many people don't care about uniqueness, they want there best tools for their needs. The main value of (server) Linux is speed and efficiency, and systemd boots and manages systems and services significantly faster and more efficiently than the alternatives. Noone wants to deal with bad software for the sake of heterogeneity and there's nothing with converging on the best solution to a shared problem.

[@George-lt6jy]

install gentoo

[@xnoreq]

Yes, Gentoo supports systemd.

[@anthonyvays5786]

based

[@gavin9715]

Fuck off gentoo is for losers

[@ishanagarwal475]

*with systemd and gnome

[@kcireorenom8430]

@puchenyaka cgroups is a plus.

[@Zmej420BlazeIt]

"swaps are on their way out. Should we use swaps?" "You should." "You should?" "On SSDs should" "On SSDs you should." "Oh okay... But... Yeah I'm not sure we have to figure that out that swap file thing and that partition thing" *drinks awkwardly* That's Lennart

[@xnoreq]

No, he said the [requirement of automatically resizing] swap files hasn't come up and a swap partition is handled correctly [like any other partition]. There's a reason this requirement hasn't come up for swap files ... because it makes little sense. The reason to create swap files in the first place is to limit their size and put them beside other files onto a filesystem. So they shouldn't be automatically grown.

[@jaketus]

All DNS-related stuff around 1:25:00 onwards sounded terrible from the point of security. 1) DNS caching by default on machine can be side-channel attack. I mean, you can anyways do this on network-level, but the latency-variance is much higher than a local dns-server giving the answer. 2) Sounds like awesome DNS-query leakage. Consider: You're in cafe, connected to company VPN. Everybody can hear what you're searching and as DNS has non-existent security, hijack the traffic. I'm not saying I don't have local dnsmasq on my laptop with multiple DNS servers configured, but I'm aware of the risks, it shouldn't be default. On my laptop I have configured per-domain dns-servers, making certain queries always go to DNS server on the other side of VPN, certain queries going directly to other DNS servers and some queries going to local dns servers. Also I'm scared if the systemd implementation is trying to be to smart with dns, setting nameserver 127.0.0.1 won't work no longer.

[@xnoreq]

Timestamp is 1:28:50. systemd-resolvd is much more secure than the glibc implementation. Your point 1 is nonsense. See DNSSEC, DNS over TLS, DNS over HTTPS ... This has nothing to do with systemd-resolvd but the DNS protocol itself. Your point 2: ditto. And if you're connected to your company VPN then you're most likely using your company's DNS server with all traffic encrypted through the VPN tunnel. As for your last point, systemd-resolvd is a very simple forwarding DNS client. If you need anything more advanced than forwarding to a DNS server then you should pick another tool.

[@jaketus]

@@xnoreq I wasn't talking strictly of spoofing on the 1st point. I was talking about side-channel sniffing as different processes can use timing to detect each another and potentially find what the other one is doing. Also it might open ways to poison the cache, if it doesn't forcefully purge often enough (should probably do with every new connection). I'm thinking of rogue AP with login page, making redirect to a target domain while returning spoofed response with very high TTL. With system-wide DNS caching it would allow attacking other applications through browser by making the browser query a domain while poisoning the response. And you're missing the point of the 2nd. The fact that if the systemd implementation uses all available DNS servers as Lennart told at 1:30:15, meaning even if you're connected to company VPN, the systemd-resolv queries the local (potentially insecure) dns-server as well. This opens ways to sniff the company infrastructure and potentially simply hijack traffic. Practically your machine will be a beacon broadcasting everything you're doing in company network to the whole cafe-wifi. And yes, all my laptops do have local dns-server (dnsmasq) running with per-domain dns-servers set up, but I don't think it should be the default due to all the downsides.

[@xnoreq]

@@jaketus Regarding your first point, what crazy scenario are you dreaming up here? One where you don't have control over the processes on your own system? Or do you seriously intend to run a single systemd-resolved instance shared across processes that should be highly isolated from each other? Your basically inventing scenarios here for which most OS' default configuration will completely fail anyway as a means to attack systemd, even though it can be configured differently in seconds? The defaults are there for the average Joe.. Your cache poisoning example is again a weakness of plain DNS. If you use plain DNS then you've already lost as soon as you connect to any network that you don't control yourself. Regardless of the client.

[@jaketus]

@@xnoreq Sure the plan is to have control of all of your processes, but processes can be exploited especially with systems connected to internet. Past couple years the exploits with largest hype have been exactly exploits where you sniff data of another process via by tricking another to leak information. Spectre/Meltdown and co. ring a bell? If there isn't reason to NOT isolate processes, the processes should be isolated as far as practical. And with the poisoning, I'd bet 99.99% of computers on the net use plain text DNS. It is not ideal for sure, but it's the real world we live in. And yes, for your online-banking, reliable DNS is in no means (should) the mainline of defense. But many legacy protocols aren't secure. If you have (in worst case) a persistent (across boots or connections) system-wide cache on your machine, poisoning it will be trivial. Even if you couldn't attack the data itself (for example HTTPS-traffic with competent user at the end of browser), you could easily track the user, track the application and deny service. By combining different attacks, it makes dns-cache poisoning a truly powerful weapon. One could for example spoof NTP (which is usually unsecure) to turn back the date by few years to combine it with self-generated SHA1 (via collisions) certificate to a banking site. And even if you couldn't spoof any protocol/application to do anything malicious, the fact that systemd-resolv broadcasts to everybody in insecure wifi what you do at the end of a VPN surely isn't what people want. And this is where it comes to defaults. They are meant for the average Joe. By default they should be as secure as practically possible. Sure Lennart wants his DNS to resolve both RH and local DNS-names for easy access to resources on both networks. But the average Joe wants to either browse the net "normally" or use VPN for security with DNS traffic going only through the VPN. The average Joe doesn't care for resolving local names while he uses his/her VPN if he/she ever uses local names. Why would the average Joe connect between his/her own machines, everything for average Joe is on the cloud now. But make the average Joes machine broadcast to everybody what he/she does while on VPN and opening the "secure vpn" up for spoofing by the local dns-server, that's the average Joe doesn't like.

[@jaketus]

@@xnoreq And to add to the point, it was just 2 years ago when I found a serious flaw in the security of the second largest bank in my home country. Long story short; let's say their name is "bigbank.com". They had just launched a new website with subdomain "new.bigbank.com". The way they did this was to have a 302 redirect from the bigbank.com -> new.bigbank.com. Also the online-banking moved to the new subdomain. The issue was; after the launch of the new site, bigbank.com-domain was no longer available with ssl (the port 443 wasn't even open), meaning all redirect request were insecure. They also didn't have HSTS or DNSKEY set up (bunch of amateurs for sure). This means, anybody who controlled network or had the skills to hijack a network (basically anybody with some computer skills), could hijack the traffic to the online-banking site and pass it through their own proxy, replacing account numbers on wire-transfers. Now combine this with potential systemd-resolve with persistent system-wide dns-cache. You could spoof the IP of "bigbank.com" and give it ridiculously high TTL, like 136 years (the maximum of DNS standard) and the user wouldn't notice anything, as long as server at your IP would do the exact same redirect as the real "bigbank.com". Do this enough times on public wifis, poison enough dns-caches. Then one day, instead of (302) redirect, proxy the connections with modifications. You could steal plenty of money before anybody noticed that something was up. Without caching, sure in this case, you could intercept some transfers, but not as many, as you'd be limited with the number of users you can concurrently spoof the DNS to. HTTPS wouldn't help, if you just registered for example "newbigbank.com", people would think it's completely legit with the redirect from bigbank.com. Oh, and it took the bank around 4 months to fix this issue.

[@NillKitty]

His voice is a cross between Elon Musk and Elton John.

[@RogerBarraud]

NYC RedHat guy sounds like Kermit.

[@GinTonicFPV]

Looks that way too

[@deadeye1982a]

I like systemd too.

[@desktorp]

20:51 translation: iptables is in Poettering's crosshairs and he's ready to pull the trigger

[@xnoreq]

Not at all. Iptables is a completely different project with a completely different use-case. Besides, iptables has been superseded by nftables.

[@Zmej420BlazeIt]

I didn't hear him that way, what I heard was, "we are pushing the kernel developers to allow userspace to have access to a new virtual machine the kernel provides, and for some silly reason they think it's a bad idea but we will bully them until they accept it"

[@xnoreq]

@@Zmej420BlazeIt Well you don't know what you're talking about then. BPF is quite old and also quite limiting. Every program is analyzed before it is loaded and rejected if it harms the kernel. For example, you cannot do endless loops.

[@StefanNoack]

@@xnoreq So they have solved the halting problem then? :D

[@xnoreq]

@@StefanNoack How is it relevant to limited length, time and memory conditions?

[@laughingvampire7555]

the NSA employee

[@guyonearth]

Systemd: Revenge of the nerds.

[@maxsnts]

So... basically Linux now has a firewall that looks like the windows one?!! Let the flame wars begin! :)

[@xnoreq]

No, what he presented is very different from what Windows does. Btw, iptables has supported filtering based on processes for ages. Even what he's talking about was already possible for a while. Systemd just exposes this functionality with simplified unit file options.

[@maxsnts]

@@xnoreq I may be wrong but i think iptables does not allow by process. It allows by user or group and then you can tie user and process (owner), but its not by process.

[@xnoreq]

@@maxsnts Hmm, there used to be a pid match but I guess it was removed because it didn't work with SMP? So yeah, you'd have to do it by user or through namespaces.

[@maxsnts]

@@xnoreq Even that would be strange. PIDs change, how could that be part of an effective firewall rule? Anyhow, systemd has quirks, but its a nice addition overall.

[@xnoreq]

@@maxsnts People could write "wrapper" scripts that would modify iptables rules. But yeah, it's weird. Probably another reason why it was removed.

[@RogerBarraud]

24:10... which presumably will be a security auditing nightmare? :-/

[@buserror1]

About 10 years ago, Red Hat realized the only way they were going to remain viable as company was to make Linux as proprietary as possible, and do so without breaking the GPL. Whether systemd is garbage or not is beside the point; In either case, Red Hat found the right person to throw under the bus for the sake of their own greed.

[@kylegrieb2075]

24:08 "We integrate as much as we can with a life-cycle of a service"

[@doctordorkmeister5330]

So he implements a code injecting back door (rootkit) And cleverly disguised with the name rtkit and naturally Microsoft saw his value as an ethical remote communication developer.

[@mcswabin207]

Yeah. "firmly established" is what any invading force says on a hostile takeover. Thanks for confirming.

[@CMDRSweeper]

Lennart should buy an Apple and stick with that, he has only been a source of problems since he first started developing his Applified ideas for Linux. Systemd is a massive disaster that often behaves very Windowsy because of its nature and isn't natural at all to use, may be to Apple people, but guess what, those on Linux aren't there because they love Apple :D And oh my, PulseAudio, a neat concept as an idea, but oh man how much pain it has caused and it is just as stable as a castle built on a swamp with no land preparation, just a matter of time until it tumbles over and sinks into the swamp.

[@dcadfdfbdf2181]

Hey thanks for destroying VPNs on linux Lennart

[@RogerBarraud]

NO.