Kernelless Kernel Programming (eBPF) - Computerphile

The tongue-in-cheek title refers to the fact that eBPF can be a shortcut to programming inside the kernel. Dr Richard G Clegg of Queen Mary University London explains how this networking tool has morphed into more.
Richard's notes:
The Github link: github.com/richardclegg/ebpf_...
A tutorial if you want to find more:
play.instruqt.com/embed/isova...
This is written by an expert but a bit dated now:
www.brendangregg.com/blog/201...
libbpf is useful:
github.com/libbpf/libbpf
I used code from here by Brendan Gregg in my example:
github.com/iovisor/bcc
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com

[@sacredgeometry]

The king of gesticulation is back and I am here for it. What an interesting topic.

[@JosueRodriguez08]

Making me nervous

[@richardclegg8027]

"King of gesticulation" - I may put that on my CV.

[@BaalKlezmerTov]

had to minimize the screen, highly annoying looking an engineer talking like a street fraudster

[@RealEngineer]

Gesticulation tend to show enthusiasm. Enjoyed the episode 😊

[@VRchitecture]

Oh, man… have you ever seen Italians’ gesticulation while speaking?)

[@autohmae]

An example of eBPF use in production is NGINX webserver, when running as multiple processes and using just one shared UDP port for QUIC & HTTP/3, the eBPF program would route the packets to the correct NGINX webserver process. Which is a very useful optimization.

[@allanguwatudde7623]

So load balancing?

[@autohmae]

@@allanguwatudde7623 yes, a sort of loadbalancing system, the kernel already had a function for this for TCP-connections which is SO_REUSEPORT, but not for UDP and definitely not any support for QUIC. So this was a fairly simple way to do this for QUIC.

[@Verrisin]

whoa

[@bobsquaredme]

there's an HTTP/3 now?

[@fluteplayerify]

Similar to shader programming where the c code is passed as a string to the GPU driver. The GPU driver compiles, schedules, then executes the shader on whatever hardware you have.

[@richardclegg8027]

A little yes. You're targetting your code somewhere else where (for whatever reason) it will be more efficient).

[@jordixboy]

how can that be done if the code is compiled? its still code obviously (in binary) but as a string?

[@richardclegg8027]

@@jordixboy in this case it is compiled to something like byte code for a VM as Java does it.

[@stensoft]

@@jordixboy The code of the shader is not compiled, it's embedded in your (compiled) program as raw string and passed to the driver. That also means it really easy to extract the shader code from your program. In newer frameworks, it can be compiled to some intermediate binary representation which is halfway between source code and the final binary.

[@jordixboy]

nteresting, could you guys send me some article to read more about it?

[@INeedAttentionEXE]

Recently the kernel introduced eBPF for the Human interface device subsystem. In short apparently an absurdly fast way to support new mice/keyboards with many buttons. It is absolutely fascinating!!!

[@NeinStein]

Time to compile the brain driver to use thoughts as keyboard and mouse.

[@eclecticaaronbentley]

You said the compiler rejected the infinite loop, but the kernel actually contains a verifier to prevent infinite eBPF programs from loading, and from the traceback it looks like the failure happened when you tried to attach the program, not when you compiled it. Also, the failure location is given in instructions, not line numbers, so this would be a very unusual compiler error. The fact that eBOF programs are verified by the kernel before being run is an important part of the system, because it means the kernel, not a compiler, determines what kind of programs are safe to run (and the kernel can adjust that definition over time).

[@Snsee]

Great Episode! I am writing my bachelors thesis on a protocol that uses eBPF to send duplicated udp packets over an redundant path. It also deduplicates the packets with XDP at the destination, it's been really interesting. Also, BCC indeed is a bit dated as far as I read - thats why I used libbpf-rs, which I am really happy with so far. One huge plus of libbpf is that you can write portable code (they call it Compile Once - Run Everywhere) - basically it abstracts away the memory accesses. This way the changing kernel source won't affect compiled eBPF Programs.

[@sachamm]

Interesting thesis, is the idea to increase reliability of UDP?

[@Snsee]

@@sachamm Yes exactly, it's supposed to be used for control messages that require very low latency. While TCP is reliable it would also introduce head of line blocking

[@Robstafarian]

How do you like Rust?

[@richardclegg8027]

@@Snsee Presumably your competitor is multipath QUIC type proposals. Very cool ideas anyway. And yes, if I made this again I would use libbpf.

[@Snsee]

@@richardclegg8027 That's probably true, but so far the protocol is really lightweight and with eBPF the connection establishment can be done from Userland so it doesn't slow down the data path. I will need to look further into multipath quic when we get to the evaluation. Thanks for your comment!

[@RyanJ_]

Always happy to see a new Dr Clegg video!

[@sergioalejandrogonzalez9755]

Brilliant explanation of eBPF and great introduction. Thank you!

[@NomenNescio99]

I always pictured userland above the kernel and the hardware at the bottom. I will have to watch this video upside down to be able to follow along.

[@richardclegg8027]

Are you in Australia? Hah... as long as the kernel is the middle bit it works.

[@Peregringlk]

For me, the kernel is above userland. The computer is like a country and the kernel is like the goverment. It goes on top of userland for me.

[@damirahman]

eBPF is fantastic! we used it all the time in grad school. it's an absolutely essential tool for doing OS research

[@cerealpeer]

loved this episode! gave me lots of ideas!

[@canban]

Dr Clegg looks surprisingly similar to Tarantino! Interestingly I am now finishing up my thesis and at one point I was using eBPF to solve a problem. It is such a powerful tool.

[@jorgerochagualtieri]

It's an amazing topic, I think it deserves a better videos

[@ssl3546]

Patton Oswalt is such a renaissance man.

[@CSniper]

Fascinating stuff! Would be nice with another video going deeper on eBPF, looking into how we determine if a program is safe to run. And yes ... everybody loves the Fibonacci sequence hehe

[@as-qh1qq]

"Let's imagine you are working on..Linux - which everyone should be" - truer words were never apoken

[@n0kodoko143]

Really cool stuff!

[@paulhetherington3854]

Induct molz~tmp prnt LN''

[@yega3k]

When I first looked up “eBPF”, the “packet filtering” bit threw me off 😅. Explaining how this went from a simple packet filter to what eBPF is now was very helpful.

[@sachamm]

Super interesting, I've never heard of this before. I see in the comments that there is even an intermediate compilation standard. Presumably that's something like JVM bytecode but for a kernel VM. I do have to wonder about security though. All the problems with virtual machines, now in your kernel!

[@fringefringe7282]

Great, more!

[@hayleyxyz]

That example looks very cool - I usually implement LuaJIT in my projects if i need any extensibility, but i have considered alternatives incl. BPF. It's probably too low level for my needs, but I may play around the example as I've never used BPF before.

[@DantalionNl]

I have used eBPF for my master thesis (CS) and while eBPF is extremely interesting for including project extensibility it has very serious limitations. The most important for general extensibility is that it has no floating point support. That said for program extensibility there are ready to go userspace eBPF VMs that you can easily include the most prominent is known as uBPF (I can't share links those comments get shadow banned on youtube).

[@hayleyxyz]

​​@@DantalionNl sounds interesting, thank you 😊 dw about links, I found the uBPF project page.

[@TalalAEdwan]

Very nice 👌

[@butteredcrumpet]

I am very tired/Jetlagged and thought I was watching a Ronnie Barker video for a second there!

[@cyboticIndustries]

Its Fletch from Porridge! LEGEND! Nice to see he's finally going straight... 😂

[@marsovac]

The only thing more sustantial than the content of the video is the gesticulation :D

[@richardclegg8027]

What can I say... can't talk without waving my hands. 😂

[@Iaotle]

This is kind of missing the part of the video where he actually runs the infinite loop and sees what happens to the computer when a rogue kernel process actually runs... Still very interesting video, as someone who worked with userspace networking code (kind of going in the other direction with RDMA - moving all the TCP processing off to the NIC and utilizing userspace to do the rest), this BPF stuff is pretty neat!

[@DantalionNl]

You can not run an infinite loop with eBPF on Linux, all eBPF kernels need to be statically verifiable, you can not even use arguments for loop termination conditions. If the program is deemed not safe for execution the Linux kernel will refuse to run it.

[@DantalionNl]

@@h..hThere will be some configurable limit, typically not defined in loop iterations but in a amortized 'computational complexity' cost estimate based on the amount data moved, the calls made and the loop iterations as well as the number of branches

[@MasterHyperionMC]

@@h..hSo it used to be you couldn’t even have bounded loops, but kernel version 5.3 allowed that. The big limit now is a maximum of 1 million instructions. The reason bounded loops are allowed is because the eBPF verifier will be able to expand the loop to determine the number of instructions it will take. Of course, if you’re writing a program to run on kernel versions lower than 5.3, you can’t even have bounded loops, which I can say from experience is incredibly limiting

[@sacredgeometry]

I think he wasnt expecting it to catch it

[@Iaotle]

@@DantalionNlYeah but he literally managed to bypass the infinite loop check from the compiler didn't he? It's in the video.

[@MatthewPherigo]

Justine Tunney used eBPF to make really easy-to-use process-level sandboxing on Linux, inspired by OpenBSD's pledge/unveil syscalls. I would paste a link but Google would eat my comment.

[@uis246]

You mean seccomp?

[@killpidone]

You could make a cruel prank where whatever file is open, you could randomly inject extra characters that is returned to the user

[@richardclegg8027]

Love the idea. Or when someone does an ls to find the file you move it.

[@killpidone]

@@richardclegg8027 or write it to whatever is the equivalent of /dev/dsp nowdays

[@izimsi]

I'm getting bad flashbacks seeing BPF, XDP and other stuff like this, did some work with that and it was a real pain, although mostly because of the custom kernel and NIC drivers.

[@DantalionNl]

It used to be particularly difficult to get working right but there has been a lot of consolidation of the number of available runtime frameworks, the runtimes are typically incompatible with one another making the experience pretty hard to understand and use. Especially now that LLVM can compile bytecode and that we have BTF, a system to help make the bytecode work regardless of Linux kernel version, things have settled quite a bit.

[@richardclegg8027]

I didn't find it too awful. I learned to code it for this video. Compared to (say) DPDK it is low effort to learn.

[@izimsi]

@@richardclegg8027 well I was pairing it with dpdk but only for development purposes

[@j-mr-t4025]

Importantly, the kernel verifier not only prevents guaranteed infinite loops, it also prevents any loops which can't be proven to be finite. That might seem like a trivial difference, but it means that the subset of "runnable" eBPF (as opposed to all compilable eBPF) is also not Turing complete, because you can't prove whether or not an arbitrary Turing machine holds, so if we only allow turing machines for which this is possible (i.e. equivalent automata of a higher Chomsky Type), we've excluded some possible programs. I would have really liked a more in-depth look into eBPF bytecode in the video for that reason. As far as I know the verifier checks the programs control flow graph, and if that is not acyclic, rejects the program (consequently loops have to be unrolled by the compiler to be able to pass the verification). But I would have liked to know a bit more about the verification process, and what additionally goes into it. I'm still glad you've shown this demo, thank you!

[@lepidoptera9337]

What stops me from writing acyclic code that will take longer than the lifetime of the universe to terminate? The entire idea sounds naive to me. Unless they are setting a low (1 minute) limit on top, without the ability to restart... and even then one can probably still do some rather malicious things with it.

[@steubens7]

it's still c promotional rules, bpf has a known word size, untyped variables are ints, changing it to u64 is probably the same resulting type

[@richardclegg8027]

The verifier runs C promotional rules? Even weirder then it changes the outcome.

[@Richardincancale]

Looks like a great tool for hackers wanting to implement persistent attacks!

[@AyushTechnoholic]

😂

[@DantalionNl]

A common topic during blackhat and defcon for the last years indeed, although making them undetectable and persistent is actually particularly hard. Especially since the events registered can be easily queried and the events are cleared every time the system reboots.

[@richardclegg8027]

I had not thought too much about it when I was asked. *But* if I am running some code a hacker wrote then I already kind of lost right. :)

[@nixonkutz3018]

Very informative - lots going on under the covers. A rich potential attack vector? Or not, if the eBPF code has to run as root?

[@StefanReich]

You can do anything as root

[@lepidoptera9337]

@@StefanReich Which is just as bad an idea as this. A user machine should NEVER allow root access.

[@tablettablete186]

​@@lepidoptera9337So, sudo shouldn't be allowed under any circumstances?

[@lepidoptera9337]

@@tablettablete186 sudo is a naive workaround for the failed "root" concept. The average user should NEVER have to use it. Look, all of this was invented at a time when computer security was not a problem. Today it is. Nobody was doing their bank transactions on a multi-user mainframe in the 1960s and early 1970s. Except for the bank, that is... and they could physically secure their hardware and the only possible "hackers" were their own employees who had hardware access. That made these simple minded concepts somewhat workable. They are not workable today.

[@jimmorrison6177]

This is so weird, I just learned about eBPF like a week ago and got really interested in it and started writing some toy programs with it to get a feel for it. So seeing a Computerphile video uploaded just now is kinda creepy.

[@dipi71]

I build my own kernels, and I never include anything BPF in them. Same goes for IPv6, Bluetooth, WiFi, initrd and so on. All of a sudden the kernel is a mere 5 MB - instead 100MB plus kernel objects plus initrd.

[@spiral09]

So if a program require a kernel feature but my kernel isn't compiled with that feature I could use eBPF to essentially run the program without the need of recompiling the kernel with necessary feature?

[@DantalionNl]

Mostly no, eBPF programs (in Linux) can only run as events based on what the kernel is doing so whenever a packet arrives on a specific interface. However, you could never run it as a driver for the network interface in the first place. So similarly you can register events on filesystem operations but can't implement the filesystem itself.

[@richardclegg8027]

Sort of depends what you mean by feature. If your kernel is not monitoring something at all you can't make it do so using eBPF. But if you want a higher level feature that you can derive from other features then you could in theory.

[@fabianmartin88]

Why not load the custom kernel module for this?

[@jms019]

The correct ways to monitor use of a kernel function on Linux is ftrace. Ideally systemtap would work but in my experience it doesn’t. Dtrace on the other has always worked as documented (on those platforms). For general code write a module.

[@richardclegg8027]

Module is kind of a half way house but it is a lot if all you want to do is (say) write a little monitoring program. Plus now your user needs a module installed for your monitor and then they need another module for the next person's monitor and so on and so on... pretty soon they got ten extra modules in their kernel.

[@bosstowndynamics5488]

Not a computer scientist, but as far as I can tell this is less about monitoring the use of the kernel and more about using the kernel to monitor something external to the kernel, eg a hardware interface of some sort. This kind of makes sense given that it's built out from a network interface monitor, and certainly to say that it's "incorrect" when it's broadly used in practice would seem to render the term "incorrect" somewhat meaningless.

[@as-qh1qq]

It can detect infinite loops - that's insane

[@kardeef33317]

Is this the same as system hooks in windows?

[@matthewbooth4612]

Would love to see a video on this by somebody who really knows the topic.

[@yadukrishnan4174]

What is that drawing on his wall?

[@gcm4312]

Why would a fibonacci number with N larger than 40 overflow? He was using 64 bit unsgined integers so the limit would be over 18446744073709551615, which is between N=92 and N=93

[@miochan8875]

I am interested in learning kernel programming.Can anyone suggest where to begin.

[@execute214]

nice video!

[@execute214]

@NumberOneRated1997 haha, you are quite a funny guy (not saying it in a sarcastic way).

[@thealliedhacker]

17:00 -- So basically the same as everything else on Linux then?

[@Verrisin]

ok, but if it's so limited, why this weird C syntax ? - Why not something simple? Or Zig like even? (that has built in maximal stack depth, etc.)

[@Apithia]

What if you modify the compiler to allow malicious code?

[@memchk]

Doesn't matter, the kernel itself does the final verification on the compiled eBPF bytecode.

[@DantalionNl]

The Linux eBPF verifier will determine that your program is not statically verifiable and not run it. All eBPF programs need to be statically verifiable by the Linux kernel prior to execution. The static verification also checks for things like guaranteed termination of the program, so you can't use arguments in loop counters for instance.

[@funnygeeks8126]

​@DantalionNl the verification isn't to stop malicious code, it's to stop you from bricking your OS by accidentally looping forever in kernel space. You need yo be root or have the CAP_BPF capability to inject eBPF code. It's assumed you will only be granted that permission if you're a trusted user.

[@builderk]

One minor correction at 6:04: I think you're referring to XDP, not XDF.

[@whtiequillBj]

wouldn't this (from your example) still run into the problem of that you can't tell if a program will run forever due to the halting problem?

[@AGENTX506]

Solving the halting problem would be having a compiler that states whether a program halts or not. What this compiler says is either 'yes, this program halts', or 'I can't say anything about whether it halts or not'. If the compiler can't say anything then the program is treated as if it was non-halting and is not allowed to run in the kernel.

[@SuviTuuliAllan]

I really hope that it doesn't become anything like what JavaScript is for the Web.

[@richardclegg8027]

Hah... I am afraid it is a common analogy. JavaScript but for kernel is how people talk about it at conference.

[@thomaslegris5996]

Given the limitations of eBPF programs, there is little risks of that happening. The only commonality eBPF bears with JavaScript is they both run on VMs, this is it. eBPF is more like the kernel WASM rather.

[@robsku1]

@@richardclegg8027 What an awful analogy - I wish they didn't. I'm glad I managed to learn about this before I heard the analogy - well, perhaps the video mentioned it, but it showed what it _really_ was like, and it was nothing like that analogy. I hope someone at these conferences would hold a speech about why this analogy might be better left out. When I hear _"like Javascript, but for kernel",_ it immediately raises a crapton of red flags in my mind. But when I saw this video I actually thought of it as quite cool technology with potential when used the right way.

[@xTerminatorAndy]

can you do a video on io_uring please

[@allanguwatudde7623]

What implications does it have on security?

[@lepidoptera9337]

It kills it. ;-)

[@largepimping]

I love hearing the Brits, their accents are so posh!

[@esra_erimez]

How does this compare to microkernel where most things are in user space?

[@killpidone]

Latency/security tradeoff

[@dsedchenko]

This thing is designed for monitoring purposes, this is not kernel module replacement. Also, afaik microkernel modules don't run in userspace. They just isolated from kernel so you can't crash a kernel with null pointer dereference.

[@richardclegg8027]

For me the key point of the microkernel is that the machine is specialized only to do very specific things.

[@esra_erimez]

@@richardclegg8027 Agreed.

[@hayleyxyz]

I never realised NICs could run BPF

[@DantalionNl]

Not only NICs but also storage systems such as NVMe SSDs, this is known as Computational Storage and has been the topic of my masters thesis.

[@richardclegg8027]

It is pretty cool. Not sure how widespread.

[@sachamm]

@@DantalionNl Sounds super interesting. Maybe we'll see you on Computerphile one day.

[@Gooberpatrol66]

NICs can run entire linux distros

[@Luix]

So you can use Python for EBPF but is not possible with Golang

[@eclecticaaronbentley]

I am sure you can use any language with a C FFI the way Python is used here. The actual eBPF code is in C, but Rust (and apparently others) can be used. Go needs a runtime, so it can't be the actual eBPF program. (It would be too big for the verifier to accept)

[@catcatcatcatcatcatcatcatcatca]

Question: is there any reason to do this in python, or is it because the original code that used python for logic and sorting/filtering of data? Without knowing how much stuff the python class does on the background, it seems like a method to circumvent any help your IDE could offer for writing C, which sounds like a bad time. But if the python-class is doing some magic on the background, writing this in C directly could be much more code than shown here.

[@richardclegg8027]

Python is pretty easy for people on the channel to read. You could write in a lot of languages. The python part of the code does not need to be highly optimized.

[@foggy5249]

"what javascript is to the web" ... a mistake?

[@charstringetje]

The immediate 2 questions I have are: Did they solve the halting problem? And have you tried expressing recursion in terms of the Y combinator? You really need to be careful with your language design when you want to guarantee all programs in it will terminate.

[@nnewram9936]

it's a finite program

[@richardclegg8027]

It is kind of the opposite of the halting problem. Knowing what class of programs halt is hard. But specifying a subset of programs that definitely halt is easy.

[@stensoft]

You can workaround the halting problem with how static analysis does it: it's considered non-halting if it can't prove that it halts

[@framegrace1]

!?! The immediate 2 questions I have are: - What does solving the halting problem have to do with eBPF ? - Why recursion, and expressing it with the Y combinator is so important for monitoring an OS? It's just an embedded language for a very niche application, not haskell.

[@kaltwarraith5172]

DKMS is another solution, though not as portable

[@nikhilsulghur7589]

"... let's imagine you're on linux, which everybody should be..."

[@afroninjadeluxe]

TempleOS is always ring 0 baby

[@jco997]

eBPF controls the Kernel, like an arduino controls a LCD display. This type of technology should be called "Controlling the Kernel as a toy" 🤖

[@j.r.r.tolkien8724]

That white IDE theme is criminal; not only is it blindingly bright but also regular code and comments have the same color 🤣

[@magnuswright5572]

That's because the whole program is a string literal, so it's not formatted

[@richardclegg8027]

Exactly as Magnus below said. The IDE correctly colours the "code" in C as if it is a string because it is a string. It colours the python correctly.

[@TheGreatAtario]

I think there's a guy whom the devs can ask about whether their "infinite loop detection" system can ever work. Name was Kurt Gödel, if I remember correctly.

[@pv2b]

Wait, you're telling me that the developers of BPF haven't managed to solve the halting problem?

[@richardclegg8027]

They don't need to solve the halting problem. They need to specify a subset of programs that definitely halt. It is kind of the opposite of the halting problem. They rule out some program which would be fine to run and would halt in time.

[@pv2b]

​@@richardclegg8027 Right, that makes sense. Although even that couldn't prevent an algorithm that eventually halts but does so after an inordinate amount of time. Unless... maybe if you constrain what kinds of loops are allowed somehow? Or just not allow loops at all?

[@richardclegg8027]

@@pv2b I'm not honestly sure at that point. If you want to look it up you can look up the eBPF verifier. I thought about the same myself before making this but the video was getting long already.

[@tomaszkoczko8606]

Gee .. Linux developers discovered DTrace😄

[@huppsy]

bro is zesty

[@D1ndo]

Wow, so this is a fully fledged Linux backdoor, then? How can this be safe at all? Surely there must be exploits that can bypass whatever heuristic they have that prevents compiling and running malicious code.

[@alpo789]

How is this a backdoor? You must be root to insert any ebpf code, at that point you hahve full control anyway.

[@lepidoptera9337]

@@alpo789 I never understood why people think that "root" is safe. For 99% of users it's more likely that the hacker knows how to become root before the user does.

[@anon_y_mousse]

This isn't kernel-less as it's still very much using the kernel.

[@GilesBathgate]

ChatGPT thinks they should call it KernelScript.

[@pankajkhushalani]

eBPF IS THE BEST THING EVER BUT WITH THE WORST DEVELOPMENT EXPERIENCE 😩😩 eBPF appeals to the masochist in me

[@richardclegg8027]

Hah... a bit extreme. Some technology is far harder to get a working build going. But yes, it was not quite as easy as I hoped.

[@zweitekonto9654]

Ah yes the human utopian dream. Doing something without actually doing it.

[@JuliusUnique]

first

[@hayleyxyz]

​​@NumberOneRated1997 what is wrong with you

[@hayleyxyz]

​​@NumberOneRated1997 KZbin isn't the place for creepy roleplaying. Maybe find some friends online to do that with so you aren't spamming up someone's channel, yeah?

[@the_original_dude]

Why is someone who doesn't know much about eBPF is explaining it to other people?

[@lepidoptera9337]

Because this is the internet? ;-)

[@KipIngram]

Too slow, man - I'm up to 3:40 and you still haven't gotten to the POINT of what eBPF actually is. You just made me curious, so I've gone now to look it up elsewhere...

[@zxuiji]

So in short it's a MAJOR security flaw that needs to be patched pronto

[@berkayuner]

Actually, it is used to make the systems more secure than not.